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ABSTRACT 


This  thesis  was  an  attempt  to  construct  and  refine  a 
model  procedure  for  DoD  contracting  activities  to  utilize  when 
integrating  quality  factors  such  as  Total  Quality  Management 
into  the  source  selection  process  for  major  acquisitions.  The 
primary  objective  of  this  thesis  was  to  modify  application 
guidelines  for  the  Malcolm  Baldridge  Quality  Award  into  source 
selection  criteria,  and  then  devise  a  procedure  to  temper  the 
results  of  the  basic  proposal  evaluation  with  a  degree  of  risk 
determined  from  a  Performance  Risk  Assessment  of  the  offeror. 

A  secondary  objective  of  this  thesis  was  to  construct  the 
model  procedure  in  a  manner  which  allowed  maximum  flexibility 
to  contracting  activities  to  tailor  any  aspects  of  the 
procedure  to  fit  local  requirements,  regulations  and  standard 
operating  procedures. 

The  final  objective  of  this  thesis  was  to  obtain  feedback 
from  knowledgeable  and  experienced  Government  contracting  or 
policy  personnel,  and  to  modify  or  refine  the  procedure  into 
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I.  INTRODUCTION 


A.  AREA  OF  RESEARCH 

The  researcher  will  identify  and  examine  the  elements  of 
quality  in  the  contractor  performance  of  Department  of  Defense 
(DoD)  contracts.  Once  identified,  the  most  salient  and 
quantifiable  quality  factors  will  be  incorporated  into  a  model 
Source  Selection  Evaluation  Procedure  for  use  by  Procurement 
Contracting  Officers  (PCO's)  and  other  source  selection 
personnel  involved  in  formal  source  selection.  The  researcher 
will  then  examine  various  sources  of  contractor  present  and 
past  performance  and  a  recommended  Performance  Risk  Assessment 
(PRA)  procedure  from  the  Army,  and  incorporate  it  into  a 
procedure  for  Source  Selection  Evaluation  Boards  (SSEB's)  or 
Source  Selection  Authorities  (SSA's)  to  verify  and  validate 
offered  contractor  quality  information  with  actual  contractor 
performance  quality  history. 

B .  RESEARCH  QUESTIONS 

1 .  Primary  Research  Question 

What  would  be  a  useful  and  practical  Source  Selection 
Evaluation  Procedure  model  for  PCO' s  and  SSEB's  which 
would  incorporate  significant  and  quantifiable  quality  factors 
with  a  procedure  to  verify  and  validate  offered  contractor 
quality  information  with  contractor  quality  history? 
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2  .  Subsidiary  Research  Questions 

1.  How  are  Packard  Conjnission  findings  and 
recommendations  and  the  DOD-wide  TQM  implementation 
efforts  spawning  attempts  to  incorporate  quality 
considerations  into  DoD  source  selection  evaluation  methods? 

2.  How  did  CICA  impact  DoD  use  of  Best  Value  source 
selection  procedures? 

3.  What  are  the  current  attitudes  of  Navy  PCO's,  PM' s 
and  commercial  procurement  personnel  towards  quality  factors 
in  source  selection  procedures? 

4.  Which  quality  factors  are  considered  as  the  most 
critical  and  quantifiable  to  Navy  PCO's  today? 

5.  What  are  the  various  systems  throughout  DOD  which 
collect  and  disseminate  historical  contractor  quality  history 
data,  and  can  be  utilized  in  the  conduct  of  a  Performance  Risk 
Assessment  (PRA) ? 

C .  OBJECTIVE 

The  objective  of  this  research  is  to  propose  a  source 
selection  method  which  addresses  one  of  the  recommendations  of 
the  Packard  Commission  as  well  as  to  possibly  aid  in  the 
implementation  of  Total  Quality  Management  in  DoD  by 
constructing  a  model  Source  Selection  Evaluation  Procedure 
which  can  add  quality  factors  and  TQM  to  existing  technical 
and  cost  considerations.  The  information  which  the  model 
quality  factors  request  in  solicitations  will  then  be 


2 


validated  and  verified  by  the  SSEB  against  contractor  quality 
history  as  reflected  in  centralized  past  performance  data 
collection  bases  and  other  sources  of  past  performance 
quality.  Therefore,  if  several  contractors  are  relatively 
equal  or  close  in  technical  abilities  and  cost,  the  quality 
factor  could  be  a  valid  tie  breaker,  depending  on  how  this 
model  is  used  or  how  much  weight  is  placed  on  quality 
considerations . 

D.  SCOPE  OF  THE  THESIS 

This  study  will  be  specifically  limited  to  procurement 
actions  of  such  magnitude  or  importance  as  to  require  formal 
source  selection  procedures,  with  an  emphasis  on  the 
acquisition  of  major  weapons  systems  on  a  greatest  or  best 
value  basis.  The  background  research  will  concentrate  on  the 
period  1984-1991,  the  period  just  after  the  implementation  of 
CICA  to  the  present.  The  quality  factors,  especially  TQM,  will 
be  drawn  from  the  application  guidelines  for  the  Malcolm 
Baldridge  National  Quality  Award.  The  most  current  and 
comprehensive  service/agency  initiatives  for  collecting  and 
disseminating  contractor  quality  information  will  also  be 
examined  as  part  of  the  Performance  Risk  Analysis  portion  of 
the  model . 
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E .  METHODOLOGY 


This  study  will  consist  of  three  phases.  The  first  phase 
all'll  combine  extensive  literature  review  and  research  with 
personal  interviews  from  appropriate  DOD  and  private  sector 
personnel.  This  phase  will  be  finished  upon  com.pletion  of  a 
model  Source  Selection  Evaluation  Procedure,  and  procedures  to 
verify  the  offered  contractor  information. 

The  second  phase  will  consist  of  the  distribution  of  this 
model  and  procedures  to  selected  DoD  and  private  sector 
personnel  for  critical  review  and  comment.  The  selection 
process  will  not  be  random,  stratified  or  in  any  way 
scientifically  structured,  but  will  be  centered  around 
selection  of  knowledgeable,  experienced  individuals  who  can 
objectively  review  and  analyze  the  model  for  possible 
improvements  or  refinements. 

The  third  phase  will  consist  of  adjusting  or  modifying  the 
model  and  procedures  with  valid  inputs  from  the  selected 
experts.  The  resulting  product  should  be  more  flexible  and 
adaptable  for  possible  use  by  PCO's  and  SSEB's  in  DoD 
contracting  activities. 

F.  ORGANIZATION 

The  remainder  of  the  thesis  is  organized  into  the 
following  chapters: 

1.  Chapter  II  will  provide  background  information  on 
and  a  description  of  quality,  on  what  Total 
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Quality  Management  is,  what  the  current  attitude 
towards  quality  is  in  the  Navy  as  a  source 
selection  evaluation  factor  in  procurements,  how 
the  FAR  addresses  quality  in  source  selection,  and 
what  are  the  primary  forces  influencing  quality 
perceptions  and  utilization  in  DOD  since  CICA.  A 
description  of  the  Army's  use  of  TQM  and  other 
continuous  process  improvement  processes  as 
evaluation  and  performance  risk  assessment  factors 
in  an  actual  major  procurement  will  also  be 
presented . 

2.  Chapter  III  will  be  a  model  Source  Selection 
Evaluation  Procedure  integrating  TQM  and  past 
performance  into  a  feasible  method  for  DoD 
contracting  activities. 

3.  Chapter  IV  will  examine  performance  risk,  will 
address  the  legal  aspects  of  utilizing  past 
perfoi^mance  data  in  performance  risk,  will 
describe  the  Performance  Risk  Assessment  procedure 
as  it  is  generally  conducted  in  DoD,  and  will  list 
and  describe  some  of  the  various  sources  of  past 
performance  data  available  to  DoD  procurement 
officials  today. 

A.  Chapter  V  will  synopsize  any  feedback  received 
from  reviews  and  critiques  of  the  model  plan,  and 
will  contain  any  revisions  to  the  model  based  upon 
this  feedback. 

5.  Chapter  VI  will  provide  conclusions  based  upon 
findings  and  recommendations  regarding  the 
feasibility  of  utilizing  the  model  procedure  and 
the  future  of  quality  factors  in  DoD  source 
selection.  Areas  of  study  that  warrant  further 
research  will  also  be  identified. 
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II.  BACKGROUND 


A.  INTRODUCTION 

This  chapter  will  provide  background  information  on  what 
quality  is  perceived  to  be,  what  Total  Quality  Management  is, 
what  the  current  Navy  attitude  is  towards  quality  as  a  source 
selection  evaluation  factor  in  procurements,  and  how  the 
Federal  Acquisition  Regulation  addresses  quality  in  source 
selections.  A  general  overview  of  the  primary  forces 
influencing  attitudes  toward  quality  in  DoD  source  selection 
procedures  since  CICA  implementation,  and  of  the  Army's 
utilization  of  quality  factors  in  an  actual  procurement  will 
also  be  presented. 

B.  WHAT  IS  QUALITY? 

Quality  in  products  or  services  is  not  as  easily 
quantifiable  or  describable  as  other  characteristics  such  as 
dimensions,  weight,  performance  parameters  or  cost /price.  In 
essence,  the  relative  elusiveness  of  a  concrete  definition  for 
use  in  solicitations  or  of  an  accepted  method  to  validate 
offeror  quality  claims  against  actual  past  performance  has 
made  quality  a  seldom  used  evaluation  factor  for  DoD  PCO's. 
Quality  has  been  a  heavily  used  factor  in  commercial/industry 
purchasing  offices  for  many  years,  and  there  exists  a 
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significant  difference  in  perceptions  of  quality  between  the 
Government  PCO's  and  private  purchasing  officials. 

When  searching  for  a  commonly  accepted  definition  of  ■ 
quality,  the  writings  of  four  prominent  authors  in  the  field 
of  quality  management:  Crosby,  Juran,  Doming  and  Garvin, 

stand  out  as  being  the  most  comprehensive  and  widely  accepted. 

P.B.  Crosby  defines  quality  as  simply,  "conformance  to 
requirements"  [Ref.  1:  p.  14].  His  main  point  regarding 
quality  is  that  it  provides  cost  effectiveness  and  value  as  a 
product  characteristic  because  the  costs  of  scrap,  rework, 
service,  warranty,  inspections  and  tests  which  result  from 
noncomformance  to  requirements  or  specifications  far  exceed 
the  cost  of  efforts  to  design  and  produce  products  or  services 
which  "do  not  fail  in  the  field"  [Ref.  1:  p.  15-16]. 

J.M.  Juran' s  simple  definition  of  quality  is  "fitness  for 
use"  [Ref.  2:  p.  2-2] .  The  user  or  customer  makes  this  fitness 
determination  based  upon  features  of  the  product/service  the 
user  recognises  as  beneficial. 

W.  Edwards  Deming  is  arguably  the  most  widely  known  and 
respected  author  and  practitioner  in  the  field  of  quality 
management,  in  fact  his  name  is  synonymous  with  Total  Quality 
Management  (TQM) .  He  is  credited  for  successfully  implementing 
the  total  quality  approach  and  culture  into  Japanese 
manufacturing.  He  does  not  offer  a  firm  working  definition  of 
quality,  rather  he  describes  the  difficulty  of  defining 
quality  [Ref.  3:  p.  169]: 
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The  difficulty  in  defining  quality  is  to  translate  future 
needs  of  the  user  into  measurable  characteristics,  so  that 
a  product  can  be  designed  and  turned  out  to  give 
satisfaction  at  a  price  that  the  user  will  pay.... The 
quality  of  any  product  or  service  has  many  scales.  A 
product  may  get  a  high  mark  in  the  judgement  of  the 
consumer  on  one  scale  and  a  low  mark  on  another. 

In  direct  contrast  to  the  letter  and  intent  of  the  1984 

Competition  in  Contracting  Act,  Deming  advocates  limiting  the 

number  of  suppliers  to  a  chosen  few  for  long  term 

relationships  because  [Ref.  4  :  p.  13]: 

We  can  no  longer  leave  quality  and  price  to  the  forces  of 
competition  -  not  in  today's  requirements  for  uniformity 
and  reliability.  Price  has  no  meaning  without  a  measure  of 
quality  being  purchased.  American  industry  and  the  U.S. 
Government  are  being  rooked  by  the  rules  that  award 
business  to  the  lowest  bidder. 

D.A.  Garvin  has  the  most  sophisticated  view  of  quality, 
using  five  different  approaches  for  providing  a  framework  for 
defining  quality:  the  transcendent,  product-based,  user-based, 
manufacturing-based  and  value-based  approaches.  He  adds  eight 
dimensions  of  quality  to  be  considered  by  each  approach: 
performance,  features,  reliability,  conformance,  durability, 
serviceability,  aesthetics  and  perceived  quality  [Ref.  5:  p. 
101].  According  to  his  hypothesis,  the  definition  of  quality 
is  variable,  depending  on  the  definer's  approach  point  of  view 
and  which  critical  dimensions  of  quality  were  applicable. 

From  the  discussion  above  it  is  obvious  that  there  is  no 
single  complete,  all  encompassing  definition  of  quality  which 
is  universally  recognized  and  accepted  in  industry  or 
Government,  there  are  just  too  many  variables  and  viewpoints. 
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Perhaps  the  very  best  operational  definition  of  quality  in  DoD 
today  was  offered  by  Dr.  Robert  E.  Costello,  the  former  Under 
Secretary  of  Defense  for  Acquisition,  when  he  proposed  a 
definition  of  quality  which  was  accepted  by  both  Government 
(The  Defense  Science  Board  and  The  Defense  Manufacturing 
Board)  and  industry  (The  Manufacturing  Advisory  Committee) 
[Ref .  6 :  p.  14] : 

Quality  is . conformance  to  correctly  defined 

requirements  satisfying  customer  needs. 

Dr.  Costello  expanded  further  on  this  definition  of  quality 

[Ref.  6:  p.  15]: 

. . . .the  composite  of  material  attributes,  including 
performance  characteristics  and  features  of  a  product  or 
service  to  satisfy  a  given  need.  Translation:  Is  the 
product  good,  and  will  it  do  the  job  for  which  it  is 
intended? 

C.  TOTAL  QUALITY  MANAGEMENT  (TQM) 

TQM  is  a  management  style  consisting  of  a  set  of 
techniques  and  tools  for  continuously  improving  the  degree  of 
quality  as  described  in  Section  B,  in  day-to-day  operations  at 
every  organizational  level,  in  every  area  of  responsibility, 
and  in  every  product  or  service  produced.  TQM  combines 
fundamental  management  techniques  and  strategies,  existing 
process  improvement  efforts  and  statistical  evaluation  tools 
into  a  dynamic,  disciplined  structure  focused  on  continuously 
improving  all  processes  [Ref.  7:  p.  2].  TQM  is  based  on  the 
following  elements  [Ref.  8:  p.  1]: 
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1  .  Development  of  a  culture  or  environment  committed 
to  continuous  process/product /service  improvement. 

2.  An  understanding  of  the  needs  of  the  customer,  and 
keeping  those  needs  foremost  in  the  organization's 
priorities  . 

3.  Increased  involvement  by  all  levels  in  the  work 
force . 

4.  Establishment  of  a  teamwork  approach  to  solving 

problems,  and  the  use  of  crossfunctional 

cooperation  and  dialogue. 

5.  An  understanding  that  most  problems  are  a  function 
of  process  management,  and  not  people  problems. 

6.  Making  decisions  based  on  data,  not  subjective 
inferences . 

7.  Establishment  of  a  commitment  to  quality  that  is 

pervasive  throughout  the  organization,  but 

starting  with  strong  support  from  top  management. 

8.  Stressing  that  small,  constant  improvements  are 
just  as  important  as  large-scale  innovations. 


Within  DoD,  TQM  implementation  will  focus  on  continuous 
process  improvement  of  every  facet  of  its  operations,  i.e. 
internal  operations,  weapon  system  requirements  formulation, 
design,  development,  production  planning,  source  selection, 
manufacturing,  fielding  and  follow-on  logistical  support  [Ref. 
9:  p.  ii] .  The  intent  is  for  TQM  to  evolve  beyond  being  just 
another  program  or  buzzword,  and  to  literally  change  the 
quality  culture  of  the  DoD  establishment,  its  contractors  and 
their  principle  subcontractors.  This  shift  of  quality  focus 
translates  into  preventing  defects  through  quality  being 
designed-in  and  built-in  to  the  product  or  service  instead  of 
being  " inspected-in"  or  through  defect  correction.  It  also 
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means  that  continuous  process  improvement  will  replace 
"acceptable  levels"  of  defects,  properly  defined  requirements 
will  replace  approval  of  waivers  to  conformance,  and  that  the 
emphasis  will  change  to  quality,  cost  and  schedule  instead  of 
only  cost  and  schedule  [Ref.  9:  p.  iii]. 

TQM  is  process  oriented,  and  as  such  goes  beyond  the 
traditional  DoD  quality  assurance  system  and  "after  the  fact" 
product  inspection.  Some  of  the  common  tools  used  in  the 
process  of  operating  under  TQM  include:  benchmarking,  cause 
and  effect  diagrams,  pareto  charts,  statistical  process 
control,  histograms,  input/output  analysis,  scatter  diagrams, 
concurrent  engineering,  control  charts,  work  flow  analysis, 
team  building  and  time  management  [Ref.  9:  p.  50].  When  fully 
implemented,  TQM  in  the  DoD  establishment  and  military 
industrial  base  will  encompass  the  following  principle  areas 
[Ref .  9 :  p .  iv] : 


1.  The  quality  of  management.  Effective  leadership, 
both  internally  and  at  DoD  contractor  and 
principle  subcontractor  levels.  TQM  demands 
involved,  participating  and  high  quality 
management  of  all  processes  that  add  up  to  the 
acquisition  process. 

2.  The  quality  of  processes.  Every  functional  element 
in  the  DoD  and  industry  must  continually  search 
for  process  shortcomings  and  devise  ways  to 
gradually  overcome  them  through  a  continuous 
improvement  process. 

3.  The  quality  of  the  integrated  hardware/software 
systems  and  services  provided  to  our  field  users. 
With  respect  to  product  quality,  TQM  expands  the 
definition  from  the  product  conformance  focus  to 
one  which  starts  with  the  definition  of  correct 
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requirements,  then  the  achievement  of  conformance 
to  these  requirements  through  continuous  process 
improvement,  not  exclusively  through  inspection. 

The  bottom  line  will  be  the  achievement  of  total 
user  satisfaction.  It  should  be  clear  that  total 
conformance  to  an  incorrect  requirement  results  in 
a  perfectly  incorrect  product  or  service. 

D.  THE  MALCOLM  BALDRIDGE  NATIONAL  QUALITY  AWARD 

One  set  of  guidelines  or  criteria  which  can  be  effectively 
used  to  evaluate  an  organization' s  commitment  to  and 
utilization  of  some  form  of  TQM  or  quality  assurance  in  their 
corporate  charter  or  philosophy,  and  in  their  day-to-day 
management  is  the  application  and  evaluation  guidelines  for 
the  Malcolm  Baldridge  National  Quality  Award  (MBNQA) .  These 
guidelines  were  jointly  developed  by  a  Government  and  industry 
coalition,  and  are  widely  accepted  as  being  the  most  current 
and  objective  guidelines  available  today  for  evaluating 
corporate  TQM  implementation  or  other  quality  assurance 
efforts.  An  outline  of  the  guidelines  as  of  1990  are 
contained  in  Appendix  A. 

The  MBNQA  is  an  annual  award  required  since  the  passing  of 
P.L.  100-107,  The  Malcolm  Baldridge  National  Quality 
Improvement  Act  of  1987.  The  award  process  is  managed  by  the 
U.S.  Department  of  Commerce  and  the  National  Institute  of 
Standards  and  Technology,  and  is  administered  by  the  Malcolm 
Baldridge  National  Quality  Award  Consortium,  formed  by  the 
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American  Society  for  Quality  Control  and  the  American 
Productivity  and  Quality  Center  [Ref.  10:  p.  7]. 

This  award  recognizes  any  U.S.  company  which  excels  in 
quality  management  and  quality  achievement  within  three 
categories:  manufacturing,  service  and  small  business.  This 
award  promotes  and  rewards  [Ref.  10:  p.  4]: 

1.  An  awareness  of  quality  as  an  increasingly 
important  element  in  competitiveness. 

2.  An  understanding  of  the  requirements  for  quality 
excellence . 

3.  The  sharing  of  information  on  successful  quality 
strategies  and  on  the  benefits  derived  from 
implementation  of  these  strategies. 

These  application  guidelines  are  not  only  effective  for  use  in 
gathering  information  for  MBNQA  evaluation  procedures  by  the 
Board  of  Examiners,  but  they  have  been  extremely  useful  to 
many  companies,  including  McDonnell-Douglas,  as  a  method  for 
self-diagnosis  of  the  company's  commitment  to  quality  and 
continuous  process  improvement  [Ref.  llj . 

As  a  measure  of  how  seriously  Government  and  industry 
takes  this  award,  over  180,000  applications  were  submitted  for 
the  1990  award.  The  award  has  also  received  high  level 
attention  from  the  President  of  the  United  States  to  the 
presidents  and  CEO's  of  most  firms  in  the  U.S.  today  [Ref.  10: 

p.  1  ]  : 

The  improvement  of  quality  in  products  and  the  improvement 
of  quality  in  service  -  these  are  national  priorities  as 
never  before . All  American  firms  benefit  by  having  a 
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standard  of  excellence  to  match  and  perhaps,  one  day  to 
surpass.  There  can  be  no  higher  standard  of  quality 
management  than  those  provided  by  the  winners  of  the 
Malcolm  Baldridge  National  Quality  Award. 

-  President  George  Bush 

If  you  intend  to  become  the  world's  best,  it's  essential 
to  establish  what  that  takes,  document  it,  take  action  on 
it,  and  then  measure  your  progress.  The  Malcolm  Baldridge 
is  a  road  map  for  us  to  achieve  our  vision  of  becoming  the 
world's  preeminent  space  systems  company. 

-  Ken  Francis,  President 
McDonnell-Douglas  Space  Systems  Company 


E.  THE  CURRENT  ATTITUDE  TOWARDS  QUALITY  AND  TQM  IN  THE  NAVY 

Some  of  the  key  characteristics  of  DOD  procurement 
activities  in  general  and  Navy  procurement  activities  in 
particular  are  that  the  methods  are  structured  in  nature, 
centralized  to  some  extent  to  provide  consistency,  and  open  to 
audit  and  review.  The  Navy  purchasing  environment  is  further 
impacted  by  many  factors  outside  of  the  control  of  individual 
PCO's.  These  factors  include:  socioeconomic  goals,  influence 
of  the  dynamic  federal  budget  process  with  its  fluctuations, 
the  distinction  between  price  and  cost,  the  difficulty  in 
establishing  specifications,  the  distinction  between  best, 
suitable  and  "goldplating,"  the  preference  for  fixed  price 
contracts,  and  the  strong  preference  for  competition.  Within 
this  environment,  the  PCO  must  take  written  requirements  from 
the  requiring  office  or  activity,  match  them  with  available 
suppliers,  and  negotiate  the  most  favorable  terms  for  the 
purchase.  Price  is  most  often  the  dominant  source  selection 
evaluation  factor  in  this  procurement  process,  and  may  be 
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viewed  as  stifling  PCO  creativity  and  innovation,  as  Dr.  Harry 
Page  states  [Ref.  4:  p.  42]: 

It  has  become  traditional  practice  in  government  to  write 
purchase  specifications  in  such  a  way  that  any  potential 
supplier  can  produce  the  item,  and  award  can  be  based  upon 
lowest  price. 

With  this  environment  in  mind,  what  are  the  prevailing 
attitudes  of  the  Navy  acquisition  community  towards  quality 
factors  in  the  source  selection  process? 

CDR  Charles  A.  Perkins,  SC,  USN  performed  an  extensive 
survey  of  and  personal  interviews  with  Navy  PCO's,  Navy 
Program  Managers  (PM's)  and  members  of  the  National 
Association  of  Purchasing  Management  (NAPM)  in  1988  as  part  of 
his  dissertation  research.  The  study  itself  consisted  of 
using  two  similar  questionnaires  testing  seven  hypotheses 
[Ref.  4:  p.  41].  The  first  questionnaire  was  directed  to  the 
77  designated  Navy  Program  Managers  and  was  returned  with  a 
response  rate  of  88%.  The  second  was  sent  to  517  Navy  PCO's 
and  1,163  NAPM  members  with  response  rates  of  45%  and  37% 
respectively  (statistically  significant  samples) .  Follow-up 
interviews  were  conducted  when  requested  or  when  necessary  to 
clarify  expressed  opinions. 

Nonparametric  and  exploratory  data  analysis  statistical 
tools  were  used  to  test  the  hypotheses.  The  following  are 
supported  hypotheses  and  conclusions  drawn  from  the  analysis 
of  the  collected  data  [Ref.  4:  p.  47  -  51]: 
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SUPPORTED  HYPOTHESES: 


1.  Competition  is  viewed  as  positively  influencing 
the  quality  of  products  obtained  unless  the 
competition  is  dominated  by  price  considerations. 

2.  Price  competition  is  a  more  significant  factor  in 
Navy  procurement  than  in  commercial/industrial 
purchasing . 

3.  The  effectiveness  of  the  Navy  procurement  process 
in  obtaining  quality  products  is  inadequate 
because  of  the  overemphasis  on  price  and  the  poor 
feedback  of  accurate  and  timely  quality  related 
information  from  the  end-users  back  to  the  Navy 
PCO. 


CONCLUSIONS : 


4  .  The  comparison  of  the  attitudes  and  preferences  of 
the  sample  populations  of  Navy  PCO' s  and  NAPM 
members  showed  a  significant  difference  in  the 
importance  they  place  on  quality  factors  in  their 
purchase  decisions,  with  heavier  attention  placed 
on  it  in  the  commercial/industrial  sector. 

5.  Satisfaction  of  the  minimum  specifications  drives 
the  award  in  price-based  decisions. 

6.  Consideration  of  other  non-price  related  factors 
evokes  a  notion  that  the  item  is  "goldplated, "  or 
that  too  much  is  being  paid  for  an  item. 

7.  Interviews  with  Navy  contracting  officials 
established  that  a  method  of  source  selection 
based  upon  prestated  quality  measures  would  be 
used  if  a  feasible,  generally  accepted  model  were 
available . 

8.  Of  CDR.  Perkins'  seven  quality  characteristics 
ranked  by  PCO's,  PM' s  and  NAPM's,  performance  and 
reliability  were  ranked  number  one  and  two  in 
importance  by  all  three  groups.  PCO's  ranked  the 
remaining  factors  in  order  of  importance  as 
maintainability,  schedule,  durability,  past 
performance  and  warranty  respectively.  PM's  ranked 
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them  as  durability,  maintainability,  past 
performance,  schedule  and  warranty,  and  NAPM  as 
durability,  schedule,  maintainability,  past 
performance  and  warranty  respectively. 


F.  FEDERAL  ACQUISITION  REGULATION 

FAR  subpart  15.605  addresses  source  selection  evaluation 
factors,  and  not  only  encourages  quality  to  be  an  evaluation 
factor  in  all  negotiated  contracts,  but  makes  it  and  price  the 
two  required  evaluation  factors  in  every  source  selection 
[Ref.  12 :  p.  16,  926]  : 


(B)  The  evaluation  factors  that  apply  to  an  acquisition  and 
the  relative  importance  of  those  factors  are  within  the 
broad  discretion  of  agency  acquisition  officials.  .  .  .  Quality 
also  shall  be  addressed  in  every  source  selection.  In 
evaluation  factors,  quality  may  be  expressed  in  terms  of 
technical  excellence,  management  capability,  personnel 
qualifications,  prior  experience,  personnel  qualification, 
past  performance  and  schedule  compliance. 

(C)  While  the  lowest  price  or  lowest  total  cost  to  the 
Government  is  properly  the  deciding  factor  in  many  source 
selections,  in  certain  acquisitions  the  Government  may 
select  the  source  whose  proposal  offers  the  greatest  value 
to  the  Government  in  terms  of  performance  and  other  factors. 

(D)  In  awarding  a  cost-reimbursement  contract,  the  cost 
proposal  should  not  be  controlling,  since  advance  estimates 
of  cost  may  not  be  valid  indicators  of  final  actual  costs. 
The  primary  consideration  should  be  which  offeror  can 
perform  the  contract  with  results  most  advantageous  to  the 
Government,  as  determined  by  evaluation  of  proposals 
according  to  the  established  evaluation  criteria. 


G.  THE  PRIMARY  FORCES  INFLUENCING  QUALITY  PERCEPTIONS  AND 
TQM  UTILIZATION  IN  DoD  SOURCE  SELECTION  SINCE  CICA 
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Since  the  passage  of  CICA  in  1984,  there  has  been  a  widely 
held  view  that  defense  procurement  overemphasizes  the 
importance  of  cost/price  in  source  selections.  While  CICA  did 
not  specifically  forbid  the  use  of  quality  as  an  evaluation 
factor  in  DOD  procurements,  the  manner  in  which  it  was 
implemented  by  DOD,  and  subsequent  Comptroller  General 
decisions  supporting  CICA  both  contributed  to  this  view. 

While  not  openly  criticizing  the  letter  or  intent  of  CICA, 
the  President's  Blue  Ribbon  Commission  on  Defense  Management, 
more  commonly  known  as  the  Packard  Commission,  identified 
three  problems  in  1986  with  CICA's  implementation  in  DoD  [Ref. 
4  p.  42]  : 

1.  the  interpretation  that  the  government  must  buy 
from  the  lowest  price  bidder; 

2.  the  notion  that  CICA  precludes  qualification 
criteria,  consideration  of  technical  expertise,  or 
life  cycle  costs;  and 

3.  the  resulting  focus  on  the  number  of  competitions 
rather  than  the  success  the  competition  achieves 
in  terms  of  reduced  prices  or  better  products. 


The  Packard  Commission  went  on  to  make  Recommendation  F, 
that  CICA's  full  potential  could  not  be  reached  until  [Ref. 
13:  p.  62] : 

Federal  Law  and  DOD  regulations  should  provide  for 
substantially  increased  use  of  commercial-style 
competition,  emphasizing  quality  and  established 
performance  as  well  as  price.  Commercial  procurement 
competition  simultaneously  pursues  several  related 
objectives:  attracting  the  best  qualified  suppliers, 
validating  product  performance  and  quality,  and  securing 
the  best  price.  Procurement  officers  must  be  allowed  and 
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encouraged  to . give  preference  to  suppliers  that  have 

demonstrated  quality  and  reliability,  and  to  recognize 
value  (  quality  and  price  )  based  on  a  product's 
commercial  acceptance  in  the  marketplace.  Price  should  not 
be  the  sole  determinant,  especially  for  procurement  of 
complex  systems  and  services. 

This  ideal  was  supported  by  RA.DM  Robert  Moore,  the  Navy's 

Competition  Advocate  General  [Ref  4:  p.  50]: 

The  objective  is  to... change  the  focus  of  Navy  acquisition 
away  from  lowest  price  to  a  contractor's  capability  to 
produce  quality  products. 

On  25  February  1986,  President  Reagan  signed  Executive 
Order  12552,  which  required  all  departments  of  the  federal 
government  to  improve  the  efficiency,  effectiveness  and 
quality  of  the  products  or  services  delivered  through  TQM  [Ref 
7:  p,  4].  The  President's  productivity  objectives  also  set  a 
goal  of  20%  productivity  improvement  in  appropriate  functions 
by  1992. 

In  September  1987,  Everett  Pyatt,  the  Assistant  Secretary 
of  the  Navy  (Shipbuilding  and  Logistics) ,  issued  a  policy 
memorandum  on  source  selection  procedures  within  the  Navy 
which  stated  [Ref  14]  : 

The  goal  should  be  to  define  the  quality  standards 
appropriate  to  each  requirement  and  to  communicate  them 
plainly  to  selection  officials  and  to  all  offerors.  When 
the  quality  desired  can  be  sufficiently  defined  to  assure 
success  and  proposals  can  be  evaluated  to  determine  if 
they  meet  the  specified  quality,  then  the  source  selection 
criteria  should  be:  "award  to  the  lowest  priced  (realistic 
price  for  cost  type)  technically  acceptable  offeror." 

On  18  August  1988,  then-Secretary  of  Defense  Frank 
Carlucci  announced  that  DOD  would  formally  implement  TQM 
throughout  all  DOD  activities.  He  stated  that  TQM  would 
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require  a  complete  change  in  our  traditional  approach  to  doing 

business  [Ref  7:  p.  2]  and  that: 

I  am  giving  top  priority  to  the  DoD  Total  Quality 
Management  effort  as  the  vehicle  for  attaining  continuous 
quality  improvement  in  our  operations,  and  as  a  major 
strategy  to  meet  the  President's  productivity  objectives 
under  E.O.  12552. 

Within  this  overall  effort.  Dr.  Robert  Costello,  the 

former  Under  Secretary  of  Defense  (Acquisition)  ,  presented  10 

major  items  on  his  1988  defense  acquisition  agenda  which 

included  the  goals  of:  improving  product  quality  to  reduce  the 

cost  of  defective  products  and  services  through  TQM,  and 

a'’hieving  substantial  reductions  in  the  lifecycle  cost  of 

ownership  of  weapon  systems  [Ref.  6:  p.  13].  Dr.  Costello 

provided  further  amplification  of  his  goals  [Ref.  6:  p.  14]: 

We  must  emphasize  competition  based  on  quality  as  well  as 
cost,  schedule  and  performance,  including  lowest  bid.  Our 
objectives  include:  making  quality  a  factor  in  source 
selection;  giving  extra  consideration  to  companies  whose 
products  and  services  embody  the  new  concept  of  continuous 
product  improvement...  To  implement  this  strategy,  we 
will:  encourage  contracting  officers  to  look  for  ways  to 
increase  quality  when  preparing  requests  for  proposals  and 
negotiating  contracts...  Industry  must  provide  tangible 
evidence  of  its  commitment  to  quality. 

It  is  not  only  DOD  that  is  fast  becoming  an  ardent 
proponent  of  quality  as  a  prominent  source  selection 
evaluation  factor,  industry  is  also  voicing  its  opinion  that 
quality  is  a  critical  business  issue  in  reforming  the  DOD 
acquisition  system.  The  Navy  League  Executive  Forum  identified 
this  concern  [Ref.  15]: 

The  Navy  position  regarding  the  importance  of  price  should 
be  more  flexible.  There  are  occasions  when  the  Navy's  best 
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interest  would  be  served  by  deemphasizing  price  in  the 
selection  process,  such  as  when  a  substantially  better 
quality  product  is  available  for  a  relatively  modest  price 
premium ...  The  Navy  should  develop  source  selection 
criteria  recognizing  past  performance  as  part  of  "best 
value"  and  control  the  use  of  price  as  an  overriding 
selection  cr.;.teria.  We  should  improve  the  development  of 
selection  criteria,  and  expand  programs  to  collect  and 
share  vendor  performance  data... 


H.  THE  ABMY'S  USE  OF  TQM  AND  STATISTICAL  PROCESS  CONTROL 

IN  SOURCE  SELECTION 

One  of  the  first  extensive  uses  of  TQM  and  Statistical 
Process  Control  (SPC)  factors  in  a  formal  source  selection 
procedure  for  a  major  defense  acquisition  was  completed  in 
March  1990  when  the  U.S.  Army  Communications-Electronics 
Command  (ACEC)  awarded  two  multiyear  contracts  to  ITT  and 
Varian  for  the  production  of  third  generation  night  vision 
devices  and  tubes  [Ref.  16] .  The  solicitation  was  structured 
utilizing  a  best  value  basis  for  award,  and  the  Army  agreed  to 
a  total  evaluated  price  of  $491M,  which  was  $47M  higher  than 
the  lowest  priced  combination  of  awards,  an  approximately  10% 
price  premium  paid  through  the  best  value  decision.  The  Army 
considered  night  vision  devices  to  be  the  "backbone"  of  its 
night-fighting  capability,  and  it  conducted  the  acquisition 
with  a  method  which  ensurea  that  the  Government  would  obtain 
the  most  advanced  equipment  of  the  highest  quality  with  the 
best  follow-on  logistical  support  at  the  lowest  total  cost  of 
ownership  [Ref.  16]. 


21 


There  were  four  evaluation  factors  in  Section  M  of  the 


Request  for  Proposals  (RFP) :  Price,  Technical,  Product 
Assurance  and  Test  (PA&T)  and  Production  and  Management  (P&M) . 
Price  was  more  important  than  any  one  of  the  other  three 
factors,  which  were  of  equal  weight.  Combined,  however,  the 
other  three  factors  were  significantly  more  important  than 
price.  Additionally,  the  RFP  explained  that  past  performance 
would  be  separately  evaluated  to  assist  in  determining  overall 
performance  risk  [Ref.  16]. 

In  conjunction  with  the  basic  proposal  evaluation,  SPC  was 
evaluated  as  one  of  the  five  subfactors  within  the  PA&T 
factor,  and  TQM  was  evaluated  as  one  of  four  subfactors  within 
the  P&M  factor.  In  addition,  both  TQM  and  SPC  were  also 
independently  verified  as  part  of  the  separate  past 
performance/performance  risk  assessment  (PP/PRA) .  ACEC  was 
careful  not  to  mention  specific  TQM  or  SPC  requirements  or 
methods  in  the  solicitation,  but  mentioned  that  the  Source 
Selection  Evaluation  Board  members  would  analyze  how  offerors 
would  employ  their  TQM  and  SPC  techniques  to  facilitate 
quality  performance  on  this  particular  contract.  Additionally, 
the  SSEB  would  assess  the  offeror's  management  commitment  to 
timely  production  and  delivery  of  only  the  highest  quality 
equipment  [Ref.  16]. 

ACEC  conducted  several  industrywide  briefings,  culminating 
in  formal  presolicitation  and  preproposal  conferences.  Great 
pains  were  taken  to  ensure  that  the  competitors  understood  how 
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the  evaluation  and  selection  process  would  be  conducted.  The 
competitors  were  given  ample  opportunity  to  explore  and 
resolve  any  and  all  issues  [Ref.  16] . 

ACEC' s  philosophy  was  that  a  contractor's  past  performance 
record  is  a  strong  indicator  of  its  ability  to  successfully 
perform  in  the  present  and  future.  Therefore,  the  offerors 
were  informed  that  a  detailed  review  of  particular  areas  of 
past  performance  would  be  evaluated  to  assess  performance 
risk.  The  evaluation  of  past  performance  was  conducted  by  a 
separate  evaluation  team  within  the  SSEB.  The  evaluation 
encompassed  information  provided  by  the  offerors  in  their 
proposals  as  well  as  extrinsic  data,  and  focused  on  the  extent 
to  which  the  offerors  had  previously  utilized  SPC  and  TQM 
techniques  and  the  results  attributable  to  such  efforts  [Ref. 
16]  . 

The  results  of  the  basic  proposal  evaluation  for  each 
factor  (other  than  price)  were  then  correlated  by  the  SSEB 
chairman  with  the  PRA  for  each  factor  to  provide  the  overall 
factor  ratings.  Ratings  for  the  basic  evaluation  and  the 
overall  rating  were:  outstanding,  acceptable  and  marginally 
acceptable,  and  the  PP/PRA  ratings  were:  low,  moderate  and 
high  risk.  No  numerical  scores  or  weightings  were  used,  and 
the  blending  of  the  basic  evaluation  rating  and  the  PP/PRA 
rating  into  an  overall  rating  for  each  non-price  factor  was 
somewhat  subjective.  For  example,  an  "acceptable"  basic 
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evaluation  rating  for  PA&T  may  be  raised  by  a  "low"  PP/PRA 
rating  to  produce  an  "outstanding"  overall  rating  [Ref.  16]  . 

The  award  was  immediately  protested  by  Litton  to  the 
General  Accounting  Office  (GAO)  based  on  the  issue  of 
ACEC' s  accuracy  of  the  PP/PRA  ratings  for  TQM  and  SPC.  Litton 
alleged  that  ACEC  placed  undue  emphasis  on  TQM  and  SPC  in  a 
manner  inconsistent  with  Section  M  of  the  RFP,  and  that  TQM 
and  SPC  were  undef init ized  "paper  philosophies"  for  which 
there  was  no  formal  Army  guidance.  In  ruling  in  favor  of  the 
Army,  GAO  addressed  and  rejected  each  of  the  allegations,  a 
decision  later  affirmed  by  the  Federal  District  Court. 
Consequently,  the  validity  of  the  evaluation  of  TQM  and  SPC  in 
the  DoD  source  selection  process  was  carefully  reviewed  and 
confirmed,  provided  the  source  selection  process  is  conducted 
consistent  with  the  process  described  in  the  solicitation 
[Ref.  16] . 

GAO'S  decision  contained  references  to  ITT's  mature,  well 
run  TQM  program  that  provided  a  process  to  ensure  continuous 
quality  improvements  and  delivery  of  only  the  highest  quality 
products  to  the  Government.  The  decision  also  stated  that 
ACEC's  award  determination  was  consistent  with  the  evaluation 
criteria,  and  it  was  in  the  best  interest  of  not  only  the 
Government,  but  the  soldiers  and  aviators  whose  lives  depend 
on  the  quality  of  the  goggles.  A  definitive  determination  of 
whether  this  scarce  selection  process  was  successful  or  not 
cannot  be  made  at  this  time.  No  statistical  or  quality  data 
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either  from  production  on  this  contract  or  from  user  feedback 
is  currently  available  to  verify  whether  the  10%  price  premium 
paid  was  cost  effective  [Ref.  17]. 

I.  A  DoD  INITIATIVE  TO  INTEGRATE  TQM  INTO  SOURCE  SELECTION 

In  1989,  the  former  Under  Secretary  of  Defense  for 
Acquisition,  the  Honorable  John  A.  Betti,  tasked  the  Army, 
Navy,  Air  Force  and  the  Office  of  the  Secretary  of  Defense 
(OSD)  to  form  a  Process  Action  Team  (PAT)  to  examine 
alternative  me^ns  for  integrating  TQM  into  the  source 
selection  process  for  all  of  DoD  [Ref.  7] .  The  trend  now  was 
to  not  only  support  the  implementation  of  TQM  throughout  the 
defense  establishment,  but  to  encourage  and  motivate  the 
defense  industrial  base  to  adopt  some  form  of  TQM  into  its  day 
to  day  operations. 

In  May  1990,  this  PAT  completed  development  of  draft 
guidance  which  provides  program  managers,  contracting  officers 
and  source  selection  personnel  with  practical  guidelines  for 
making  TQM  a  consideration  in  source  selection.  The  Draft 
Guide  to  Integrating  Total  Quality  Management  into  Source 
Selection  was  developed  to  foster  consistency  among 
Services/Agencies  in  integrating  TQM  into  source  selection, 
but  was  designed  to  afford  the  PCO  the  flexibility  to  tailor 
and  adopt  the  approach  to  meet  specific  program  needs  [Ref.  8: 
p.  1].  The  PAT'S  concept  is  to  accomplish  the  goal  of  taking 
into  account  offeror's  continuous  process  improvement 
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activities  by  treating  continuous  process  improvement  as  part 
of  the  SSEB' s  PRA.  The  PRA  provides  a  confidence  measure  to 
the  Source  Selection  Activity  (SSA)  of  the  offeror's  potential 
to  perform  on  the  proposed  contract. 

The  Draft  Guidance  referred  to  the  use  of  a  solicitation 
provision  based  on  a  modified  version  of  the  Malcolm  Baldridge 
National  Quality  Award  (MBNQA)  guidelines  to  obtain  the 
information  for  the  assessment.  Currently  there  is  no  modified 
MBNQA  guidelines  for  use  as  solicitation  provisions  or  as 
source  selection  evaluation  criteria  due  to  an  indefinite  halt 
in  the  finalization  of  the  Draft  guidance  into  a  formal 
instruction  [Ref  18]  . 

J.  SUMMARY 

After  discussion  of  what  quality  and  TQM  are  generally 
accepted  as  being  defined  as,  evidence  was  presented  that  the 
DoD  acquisition  community  has  the  desire,  incentive  and 
mandate  to  integrate  TQM  or  other  quality  factors  into  source 
selection.  Evidence  was  also  provided  that  the  MBNQA 
guidelines  are  a  viable  information  collection  and  assessment 
tool  to  gauge  corporate  adoption  of  some  form  of  TQM  or 
continuous  process  improvement  in  its  operations  and  products 
or  services. 

The  next  chapter  will  describe  the  DoD/Joint 
Service/Agency  PAT  Draft  Guide  in  more  detail,  and  will 
present  the  researcher's  source  selection  procedure  model. 
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The  flexible  model  will  be  a  synthesis  of  the  ACEC's  method  of 
procurement  for  the  third  generation  night  vision  devices  with 
the  evaluation  guidelines  or  criteria  suggested  by  the  PAT  in 
their  Draft  Guide. 
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III.  A  SOURCE  SELECTION  PROCEDURES  MODEL 


A.  INTRODUCTION 

Chapter  II  presented  definitions  of  quality  and  TQM,  a 
description  of  the  Malcolm  Baldridge  National  Quality  Award, 
evidence  of  willingness  among  Government  PCO's  to  use  quality 
factors  in  their  source  selection  processes,  the  successful 
use  of  TQM  and  SPC  evaluation  factors  by  the  Army  in  a  major 
acquisition,  and  a  general  overview  of  a  current  DoD/Joint 
Service/Agency  initiative  to  develop  and  promulgate  a  guide 
for  integrating  TQM  into  source  selection.  This  chapter  will 
describe  the  DoD  initiative  in  more  detail,  and  will  present 
this  researcher's  source  selection  procedures  model. 

B.  OVERVIEW  OF  THE  DRAFT  GUIDE  TO  INTEGRATING  TQM 

INTO  DoD  SOURCE  SELECTION  PROCEDURES 

The  DoD/Joint  Service/Agency  Process  Action  Team  draft 
working  paper  of  a  Guide  to  Integrating  Total  Quality 
Management  (Draft  Guide)  was  given  a  limited  distribution  in 
DoD  for  the  collection  of  constructive  feedback  from  selected 
services  and  agencies  on  May  8,  1990.  The  Draft  Guide  was 

subsequently  revised  and  distributed  for  review  and  comment 
again  on  November  5,  1990.  Appendix  B  consists  of  sections  III 
through  VII  of  the  November  5,  1990  draft  working  paper. 

Sections  I  and  II  are  the  Introduction  and  the  Concept 
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portions  respectively,  and  they  are  summarized  in  the 
remainder  of  this  section. 

The  Draft  Guide  was  developed  to  foster  consistency  among 
the  DoD  services  and  agencies  by  recommending  an  approach  for 
the  inclusion  of  TQM  factors  into  the  source  selection  process 
by  PCO' s  and  PM' s  [Ref.  19  :  p.  1] .  It  is  primarily  applicable 
to  major  defense  system  acquisitions  covered  by  DoD  Directive 
5000.1  (Major  and  Non-Major  Defense  Acquisition  Programs),  but 
may  be  used  in  any  other  DoD  acquisitions  at  the  discretion  of 
the  service  or  agency  [Ref. 19:  p.l].  The  recommended 
procedures  also  allow  PCO' s  the  flexibility  to  tailor  or  adapt 
the  concepts  to  meet  the  specific  needs  of  particular 
acquisitions . 

The  principal  rationale  for  the  development  and 
promulgation  of  the  Draft  Guide  is  that  offerors  are  most 
likely  to  provide  high  quality,  on-time,  within-cost  products 
and  services  if  they  have  viable  continuous  improvement 
efforts  backed  up  with  demonstrated  results  [Ref.  19].  The 
goal  of  the  Guide  is  to  take  into  account  the  maturity  and 
effectiveness  of  the  offeror's  continuous  process  improvement 
(CPI)  programs  as  part  of  the  source  selection  process  [Ref. 
19:  p.  2]  .  Primary  emphasis  is  placed  on  tangible  evidence  and 
verifiable  proof  that  these  CPI  vehicles  are  in  place  and 
functioning  effectively  in  time  for  the  execution  of  the 
proposed  contract,  not  just  existing  on  paper  or  brochures. 
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The  key  strategy  for  achieving  the  goals  and  intentions  of 
the  Draft  Guide  is  treating  CPI  and  offeror  past  performance 
as  part  of  the  SSEB' s  estimate  of  the  risk  involved  in  making  . 
a  contract  award  to  a  given  offeror.  An  RFP  provision  based 
primarily  on  MBNQA  guidelines  being  modified  through  a 
reduction  in  scope  and  detail  is  used  to  obtain  offeror 
information  for  this  assessment.  Also  included  in  the  risk 
assessment  is  an  evaluation  of  the  offeror's  past  and  present 
performance  record  [Ref.  19:  p.  2].  Together,  these 
evaluations  by  SSEB  teams  comprise  a  PRA,  and  the 
recommendations  of  the  SSEB  are  forwarded  to  the  SSAC  for 
further  evaluation.  The  source  selection  organization  to 

which  the  Draft  Guide  and  the  recommendations  of  this  thesis 
are  primarily  applicable  to  is  the  standard  organization  for 
most  major  weapon  system  acquisitions.  [Ref.  20]. 
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C.  SUGGESTIONS  FOR  FINALIZATION  OF  THE  DRAFT  GUIDE 

In  the  opinion  of  this  researcher,  the  Draft  Guide  could 
have  offered  more  in-depth  discussion  on  the  following  key 
areas  for  PM/PCO  guidance  or  consideration: 


1 .  Actual  examples  of  solicitation  guidelines  derived 
from  modified  MBNQA  guidelines; 

2.  specific  recommendations  for  scoring  or  rating 
techniques  or  other  quantitative  evaluation 
standards  for  the  SSEB  to  utilize  in  evaluating 
the  offers  and  extrinsic  evidence,  other  than  the 
use  of  forwarding  "narrative  comments"  to  the  SSAC 
on  offeror  strengths  and  weaknesses; 

3.  recommendations  for  a  delineation  of 

responsibilities  for  evaluation  of  the  basic 
proposal  and  for  the  conduct  of  a  PRA,  and 
references  to  specific  sources  of  extrinsic  data 
for  use  in  validating  past  performance  in  the  PRA, 
other  than:  "discussions  with  technical  and 

program  managers...  and... from  actual  performance 
data  of  products,  where  available." 


In  addition  to  the  thpee  areas  found  by  this  researcher  to 
be  in  need  of  clarification  or  amplification,  there  is  one 
procedure  in  the  Draft  Guide  which  this  researcher  recommends 
to  be  modified  based  on  the  results  of  the  Army  procurement 
described  in  Chapter  II  section  G.  This  was  one  of  the  first 
cases  where  a  service  or  agency  has  extensively  utilized  TQM 
and  SPC  as  critical  subfactors  in  a  formal  source  selection 
procedure.  ACEC's  inclusion  of  TQM  and  SPC  subfactors  in  the 
basic  proposal  evaluation  as  well  as  TQM,  SPC  and  past 
performance  areas  in  the  PRA  were  supported  by  G.A.O.  and 
Federal  District  Court  decisions  since  ACEC  was  consistent  in 
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performing  the  proposal  evaluation  and  PRA  in  accordance  with 
the  terms  in  the  RFP .  Although  the  DoD/ Joint  Service/Agency 
PAT  Draft  Guide  and  the  ACEC's  third  generation  night  vision 
device  source  selection  procedure  are  similar  in  some 
respects,  they  are  fundamentally  different  in  other  areas 
which  have  important  implications  for  the  extensive  DoD-wide 
use  of  the  Draft  Guide  in  its  current  form.  The  most  critical 
aspect  in  which  the  ACEC's  procedures  differ  from  the  PAT 
Draft  Guide  consists  of: 


4.  ACEC's  use  of  TQM,  SPC  and  past  perform.ance  as 
subfactors  set  forth  in  Section  "M"  of  the  Request 
for  Proposals  (RFP)  to  be  evaluated  and  rated 
along  with  the  other  subfactors  within  the 
Technical,  PA&T  and  P&M  factors.  In  addition,  TQM, 
SPC  and  past  performance  were  also  evaluated  in 
conjunction  with  the  PRA.  The  adjectival  rating 
from  the  basic  proposal  evaluation  was  tempered 
with  an  adjectival  PRA  rating  by  the  SSEB  Chairman 
to  produce  an  overall  rating  for  each  factor  of 
each  proposal . 


The  remainder  of  this  chapter  will  consist  of 
recommendations  for  a  revision  of  the  Draft  Guide  based  on  the 
four  areas  listed  above,  including  a  proposal  evaluation 
guideline  model  for  use  by  the  SSEB  to  evaluate  and  rate 
offeror  response  to  Section  "L"  of  the  RFP. 

D .  RECOMMENDATIONS 

1.  Appendix  C  consists  of  a  proposal  evaluation  model  for 
the  SSEB  to  utilize  when  evaluating,  rating  and  scoring  the 
sections  of  the  offered  proposals  which  address  the  CPI 
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portion  of  RFP  Section  "L"  in  Appendix  B.  It  is  sufficiently 
flexible  enough  to  allow  the  SSEB  to  use  either  a  numerical 
scoring  system,  an  adjectival  rating  system,  a  narrative 
system,  utilizing  summaries  of  strengths  and  weaknesses,  or 
some  combination  of  scores,  adjectival  ratings  or  narrative 
descriptions.  The  subordinate  subfactors  themselves  are 
derived  from  the  MBNQA  guidelines,  which  have  been: 


a.  modified  into  evaluation  criteria  with  a  DoD 
orientation  i.e.  DoD  terms,  special  interest 
areas,  and  actual  examples  of  items  to  give  credit 
for  while  evaluating. 

b.  reorganized  into  a  more  logical  sequence  for 
evaluation; 

c.  modified  to  reduce  overlapping  and  "double¬ 
weighting"  in  the  evaluation  process  through  a 
consolidation  of  related  themes; 

d.  streamlined  through  deletion  of  specific  subareas 
within  certain  guidelines  because  they  were  either 
inapplicable  to  DoD  procurement,  were  too 
subjective,  or  were  potentially  too  difficult  to 
validate  with  extrinsic  past  performance  data. 


Source  selection  personnel  must  be  aware  that  they  can, 
and  in  many  cases,  should  modify,  expand,  reduce  or  otherwise 
tailor  the  language  contained  in  the  subordinate  subfactors  of 
Appendix  C.  The  "factor"  heading  has  not  been  filled  out  in 
order  to  allow  flexibility  in  assigning  these  evaluation 
subfactors  to  factors  such  as  Technical,  Management,  Product 
Assurance  and  Test,  Production  and  Management,  Integrated 
Logistic  Support  or  whatever  the  contracting  activity's 
solicitation  organization  aictates. 
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2.  In  order  to  make  the  Appendix  C  proposal  evaluation 
model  a  useful  tool  for  the  SSEB,  Sections  III  and  IV  of  the 
Draft  Guide  should  be  modified.  The  Acquisition  Plan  (AP)  and 
Source  Selection  Plan  (SSP)  in  Section  III  of  Appendix  B 
should  be  changed  from  a  reliance  on  the  conduct  of  a  PRA 
only,  to  an  adoption  of  the  ACEC  strategy  of  making  TQM,  SPC, 
and  past  performance  rated  factors  in  the  basic  proposal 
evaluation  as  well  as  rated  factors  in  the  PRA. 

Also  in  Appendix  B,  the  sample  language  in  the  Section  IV 
Executive  Summary  and  in  Sections  ”L"  and  "M”  of  the  RFP 
should  be  modified  to  inform  the  offerors  that  the  Government 
intends  to  consider  TQM,  SPC  and  past  performance  as  not  only 
evaluation  factors  in  the  basic  proposal  evaluation,  but  as 
factors  in  a  PRA  performed  separately  and  independently  of  the 
basic  proposal  evaluation.  The  recommended  procedures  for 
conducting  the  PRA  will  be  addressed  in  Chapter  IV  of  this 
thesis . 

In  order  to  provide  an  added  degree  of  assurance  that  the 
offerors  understand  exactly  what  is  required  of  them  to  be 
fully  responsive  to  the  solicitation  requirements,  the  opening 
remarks  of  subsection  2.0  -  Specific  Information  and  Data,  of 
Section  IV,  should  be  expanded  to  include  the  following 
instructions,  which  are  modifications  of  the  guidelines  for 
MBNQA  applicants  when  filling  out  their  applications  [Ref. 
10]  : 
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a.  Read  all  of  the  solicitation  categories  and 
specific  items  within  the  categories  before 
developing  responses  to  any  of  them. 

b.  Interpret  the  meaning  of  each  specific  item  within 
the  context  of  the  main  purpose  of  the  category. 

c.  Check  related  items  to  determine  the  distinctions 
between  the  types  and  extent  of  information 
required. 

d.  Respond  to  items  with  concise,  factual  statements. 
Support  statements  with  quantitative  information 
whenever  appropriate.  Use  of  graphs,  charts,  and 
tables,  properly  labeled  and  compactly  presented 
is  strongly  encouraged.  Lengthy  narratives  not 
directly  responsive  to  the  purpose  and  substance 
of  an  item  are  discouraged.  Also,  avoid  the 
reiteration  of  the  words  and  phrases  of  the 
solicitation  items  by  using  your  own  original 
words  and  phrases. 

e.  Avoid  the  use  of  anecdotal  information  or 
information  lacking  overall  context.  An  example  is 
at  times  appropriate,  but  offerors  should  make 
clear  that  the  example  illustrates  the  larger 
point  being  made,  and  is  not  itself  the  response 
to  the  item. 

f.  Report  only  what  is  requested  in  each  item,  and 
include  only  the  types  of  information  requested. 
Make  responses  self-contained,  and  not  dependent 
upon  information  given  in  responses  to  other 
items.  However,  if  other  items  contain  information 
that  will  help  to  clarify  or  strengthen  a 
response,  and  at  the  same  time,  avoid  significant 
duplication  of  information,  provide  cross 
references . 

g.  Trend  data  are  requested  to  permit  applicants  to 
demonstrate  progress  and  to  show  that  improvements 
are  sustained.  No  minimum  period  of  time  is 
specified  for  trend  data.  Time  periods  for  trend 
data  may  span  up  to  five  years  or  more  for  some 
product,  service  or  operational  characteristics, 
but  may  be  much  shorter  in  areas  where  improvement 
efforts  have  been  established  more  recently. 
Trend  data  should  be  presented  in  graphical, 
tabular  or  other  compact  form.  Include  appropriate 
benchmarks  or  other  references  that  help  to  ensure 
the  proper  interpretation  of  data. 
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h.  Identify  responses  to  categories  and  items  within 
categories  with  the  letters  or  numbers 
corresponding  to  each  category  and  items  within 
each  category. 

The  actual  solicitation  categories  and  items  within  each 
category  listed  in  the  RFP  subsection  2.0  -  Specific 

Information  and  Data,  should  be  the  same  as  the  wording  of  the 
evaluation  factors  and  subfactors  in  Appendix  C,  instead  of 
the  wording  of  subsection  2.0  of  the  Draft  Guide.  This  is  to 
ensure  that  proposals  are  evaluated  by  the  same  criteria  that 
are  set  forth  in  the  solicitation. 

3.  The  Draft  Guide  did  not  make  specific  recommendations 
for  scoring  or  rating  techniques,  or  other  quantitative 
evaluation  standards  for  the  SSEB  to  use  while  performing  the 
basic  proposal  evaluation  or  the  PRA.  This  was  the  DoD/Joint 
Service/Agency  PAT's  intent  because  it  would  not  be  desirable 
to  standardize  the  rating/scoring  techniques  throughout  DoD. 
Maximum  flexibility  must  be  allowed  in  this  area  if  it  is  to 
be  of  value  to  DoD  services  and  agencies. 

Based  on  the  ACEC  procurement,  this  researcher  recommends 
a  specific  proposal  and  PRA  evaluarion  method  which  utilizes 
a  combination  of  the  numerical  scoring,  narrative  description 
and  adjective  grading  techniques.  Appendix  D  contains  examples 
of  proposal  and  PRA  evaluation  worksheets  utilizing  the 
Continuous  Process  Improvement  of  Processes,  Products  and 
Service  component  of  the  Quality  Assurance  of  Products  and 
Services  subfactor.  The  XYZ  Corporation  is  actually  another 
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name  for  a  real  firm  which  submitted  a  proposal  to  ACEC  in 
response  to  their  solicitation  for  the  third  generation  night 
vision  devices.  The  proposal  evaluated  in  this  example  was  the 
actual  proposal  submitted  by  this  firm.  The  maximum  points 
column  available  for  each  subfactor  item  is  filled  in  based  on 
the  relative  weighting  assigned  by  the  SSEB.  The  assessment 
percentage  is  based  on  the  overall  scoring  justification.  The 
percentage  is  based  on  the  evaluator' s  best  assessment  of  the 
offeror's  fulfillment  of  the  requirement  on  the  following 
scale  developed  by  this  researcher: 

Percentage : 


Superior 

90  - 

100% 

Acceptable 

70  - 

89% 

Marginal 

60  - 

69% 

Unsatisfactory 

0  - 

59% 

The  percentage  selected  should  be  justified  in  the  "overall 
rationale  for  assessment  %"  section  of  the  worksheet,  based  on 
a  relative  weighing  of  the  strengths  versus  weaknesses. 

Once  the  scores  from  the  evaluation  of  each  subfactor  are 
completed  for  both  the  basic  proposal  evaluation  and  the  PRA, 
the  raw  scores  should  then  be  summarized  on  a  consolidated 
tally  sheet,  added  together,  then  converted  into  an  adjectival 
grade  and  risk  level  corresponding  to  whatever  groupings  of 
factors/subfactors  the  SSEB  determines  as  logical,  feasible  or 
practical . 

An  example  of  this  is  provided  in  Appendix  E.  Here,  the 
SSEB  has  consolidated  the  subfactor/factor  scores  into  one 
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numerical  score  each  for  the  Leadership  and  Personnel 
Management  and  the  Continuous  Process  Improvement  evaluation 
factors,  although  they  could  have  grouped  them  in  any  manner 
which  they  deemed  most  useful.  The  raw  scores  are  then 
converted  into  an  adjectival  rating  and  risk  level  for  the 
basic  proposal  evaluation  and  PRA  respectively.  The  final  step 
is  the  "tempering"  of  the  basic  proposal  evaluation  rating 
with  the  corresponding  PRA  risk  level  into  an  overall  rating 
for  each  factor. 

Once  the  overall  rating  for  each  factor  is  determined,  it 
is  submitted  from  the  SSEB  to  the  SSAC  along  with  any  required 
supporting  documentation  for  inclusion  in  a  final  award 
decision  process.  The  rationale  for  the  conversion  of  raw 
scores  into  adjectival  ratings  is  that  the  numerical  scores 
may  be  more  difficult  to  justify  to  the  SSAC,  or  to  defend  in 
the  possible  event  of  a  GAO  protest  from  an  offeror  or  an 
interested  third  party. 

4.  The  specific  details  of  the  conduct  of  the  PRA  by  the 
SSEB,  including  recommended  sources  of  extrinsic  past 
performance  data  and  recommended  procedures  for  determining 
performance  risk  levels  are  contained  in  Chapter  IV. 

E .  SUMMARY 

This  chapter  described  the  DoD/ Joint  Service/Agency  PAT 
Draft  Guide  in  more  detail,  offered  suggestions  for 
modifications  to  be  made  to  the  Draft  Guide  before  its 
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possible  finalization,  and  presented  a  source  selection 
procedures  model  consisting  of  a  basic  proposal  evaluation 
integrated  with  a  PRA.  The  next  chapter  will  examine  the 
Performance  Risk  Assessment  process  more  thoroughly,  and  will 
present  some  of  the  current  sources  of  contractor  historical 
quality  history  available  in  DoD  for  the  conduct  of  PRA' s . 
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IV.  PERFORMANCE  RISK  ASSESSMENT 


A.  INTRODUCTION 

Chapter  III  and  Appendices  C,D,  and  E  presented  a  source 
selection  model  consisting  of  a  procedure  for  gathering 
offeror  information  through  RFP  language  and  solicitation 
provisions  on  an  offeror's  current  TQM  system  or  its  own 
version  of  TQM,  and  the  proposed  methods  the  offeror  intends 
to  utilize  in  applying  their  continuous  process  improvement 
program  to  the  proposed  contract.  Their  proposal  would  then  be 
evaluated  and  rated,  and  the  resulting  adjectival  rating  for 
a  TQM  related  factor  would  then  be  either  raised,  lowered  or 
remain  un^'harged  by  a  corresponding  level  of  performance  risk 
into  an  overall  rating  for  that  factor.  The  basic  evaluation 
of  the  offeror's  proposal  determines  proposal  risk,  while  the 
evaluation  of  the  actual  past  performance  of  the  offeror  on 
related  prior  contracts  or  similar  products/services 
determines  performance  risk. 

This  chapter  will:  1.  examine  performance  risk,  2.  address 
the  legal  aspects  of  utilizing  past  performance  data  in 
determining  performance  risk,  3.  describe  a  recommended  five 
step  procedure  in  performing  a  Performance  Risk  Assessment, 
and  4.  will  list  and  describe  some  of  the  various  primary  and 
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secondary  sources  of  past  performance  data  available  to  DoD 
procurement  officials  today. 


B.  PROPOSAL  VERSUS  PERFORMANCE  RISK 

When  conducting  the  source  selection  process  for  major 

weapon  system  procurements,  Research  and  Development  efforts, 

large  services  contracts  or  selected  other  non-major  purchases 

•  which  involve  complexities  or  advanced  technology  or 

technological  risk,  the  Government  must  assess  the  relative 

risks  associated  with  each  offeror  and  proposal.  These  risks 

may  be  classified  as  being  either  a  proposal  risk  or  a 

performance  risk  [Ref.  21:  p.  B-2  ]: 

Proposal  risks  are  those  associated  with  an  offeror's 
proposed  approach  in  meeting  Government  requirements. 
Proposal  risk  is  assessed  by  the  SSEB  and  is  integrated 
into  the  rating  of  each  specific  evaluation  factor  and 
subfactor. 

Performance  risks  are  those  associated  with  an  offeror's 
ability  to  perform  the  solicitation's  requirements  as 
indicated  by  that  offeror's  record  of  past  and  current 
performance.  Performance  risk  may  be  assessed  by  an  SSEB 
team  independent  of  and  distinct  from  the  team  assessing 
the  proposal  risk,  or  it  may  be  assessed  by  a  Performance 
Risk  Analysis  Group  (PRAG)  separate  from  the  SSEB. 


C.  THE  IMPORTANCE  OF  ASSESSING  PAST  PERFORMANCE  IN  A  PRA 

It  has  often  been  said  that,  whether  it  be  in  reference  to 

the  history  of  the  world  or  in  DoD  acquisition,  that  the  past 

is  prologue.  Edward  J.  Korte,  the  U.S.  Army  Materiel  Command 

Counsel  said  it  best  [Ref.  22] : 

It  has  been  said  that  history  repeats  itself,  and  that  the 
person  who  does  not  study  the  past  and  learn  its  lessons 
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will  likely  be  given  another,  often  more  painful 
opportunity  to  learn.  This  observation  is  also  true  in  the 
world  of  Government  contracts.  For  example,  a  failure  to 
adequately  address  a  contractor's  poor  performance 
invariably  haunts  our  future  procurements.  When  we  fail  to 
deal  effectively  with  a  chronic,  poor  performer,  we  are 
often  later  confronted  with  the  same  problem,  frequently 
compounded,  and  much  more  difficult  to  resolve. 

DoD  Directive  4105.62,  "Selection  of  Contractual  Sources 
for  Major  Defense  Systems"  states  that  [Ref.  21:  p.  2]"  the 
offeror' s  recent  and  relevant  past  performance  (measures  by 
such  indicators  as  quality,  timeliness,  cost,  schedule, 
operational  effectiveness  and  suitability)  may  be  considered 
in  assessing  the  probability  Of  successful  accomplishment  of 
the  proposed  effort  in  a  timely,  cost-effective  manner." 
Further  justification  for  the  assessment  of  past  performance 
in  source  selection  decisions  is  provided  in  DoDD  4105.62 
[Ref.  21:  p.  2]: 

Proposal  evaluators  must  consider  the  technical,  schedule, 
operational  readiness  and  support,  and  financial  risks 
inherent  in  a  proposal.  One  means  of  assessing  that  risk 
is  to  review  an  offeror's  recent  actual  performance  in 
relevant  areas.  Past  performance  as  an  element  of  risk 
analysis,  may  be  used  as  one  predictor  of  the  probability 
of  satisfactory  performance  on  the  proposed  program  being 
evaluated.  Evidence  of  past  performance  may  be  obtained 
from  numerous  sources,  such  as  the  offeror's  preaward 
surveys,  onsite  Government  personnel  at  a  contractor's 
facility,  field  data  collection  systems,  and  other 
procuring  activities  that  are  or  were  customers  of  the 
offeror  whose  proposal  is  being  evaluated. 

As  mentioned  in  Chapter  II  Section  F  (B) ,  the  FAR  also 
requires  that  quality  be  addressed  in  every  source  selection, 
and  specifically  that  in  evaluation  factors  [Ref.  12:  p. 
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16,  926]:  "quality  may  be  expressed  in  terms  of  . prior 

experience....  past  performance  and  schedule  compliance." 

D.  BASIC  ELEMENTS  AND  LEGAL  ASPECTS  OF  THE  PRA 

Past  performance  evaluations  in  federal  acquisitions  are 
unfortunately  neither  commonplace  nor  consistent  [Ref.  22: 
p.l].  Some  procurement  activities  hesitate  to  consider  past 
performance  for  fear  of  injecting  bias  into  the  evaluation 
process,  others  hesitate  due  to  a  general  uncertainty  in  the 
interpretation  of  the  law.  Many  other  procurement  activities 
see  the  obvious  value  in  past  performance  evaluations,  but  are 
unsure  of  the  PRA  procedure.  Those  activities  which  do 
evaluate  past  performance  rely  almost  exclusively  upon 
contractor  supplied  data  rather  than  on  independent  data 
otherwise  available  to  the  Government  [Ref.  22:  p.  1]. 

Some  of  the  hesitation  and  confusion  associated  with  past 
performance  evaluations  can  be  alleviated  by  reviewing  the 
basic  elements  and  issues  involved,  and  by  understanding  the 
legal  aspects  and  precedents. 

1 .  Past  Performance  vs  Preaward  Survey 

Past  performance  evaluations  are  not  the  same  as 
preaward  surveys,  as  each  serves  a  different  purpose.  Preaward 
surveys  are  conducted  to  determine  whether  a  contractor  is 
responsible,  or  whether  the  contractor  is  capable  of  doing  the 
job.  Past  performance  evaluations,  such  as  PRA's,  are 
conducted  as  a  part  of  the  source  selection  process  in 
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negotiated  procurements  to  determine  if  a  contractor  is 
acceptable.  The  determination  answers  the  q..estion,  will  he  do 
the  job  successfully?  [Ref.  21:  p.  1].  Preaward  surveys  are 
conducted  by  the  contract  administration  office,  and  based  on 
on-site  inspection  or  information  on  hand,  an  estimate  of  the 
offeror's  capability  to  perform  a  particular  contract  is 
determined.  The  past  performance  evaluation  is  a  very  specific 
endeavor  performed  by  the  procurement  contracting  office  to 
identify  the  degree  of  risk  associated  with  each  offeror,  and 
rely' s  solely  on  each  offeror's  track  record  on  previous 
contractual  efforts  [Ref.  21:  p.  1].  If  properly  conducted, 
the  past  performance  evaluation  and  the  preaward  survey  will 
supplement  each  other  and  provide  a  more  complete  picture  of 
an  offeror's  capability  than  either  one  could  by  itself. 

2.  Key  Features  of  a  PRA 

There  are  basically  three  key  features  to  consider 
when  performing  a  past  performance  evaluation  or  PRA  [Ref.  21: 
p.  3]  : 

1.  Extrinsic  information.  A  thorough  evaluation  of 
past  performance  will  include  evaluation  not  only 
of  information  provided  in  proposals,  but  also 
information  obtained  from  other  sources.  When  this 
is  done,  the  solicitation  should  clearly  advise 
offerors,  among  other  things,  that  the  Government 
intends  to  consider  information  outside  of  the 
proposal  (extrinsic  information) . 

2.  Separate  evaluators.  To  guard  against  injecting 
undue  bias  into  the  evaluation  process, 
reasonable/sound  practice  suggests  separation  of 
those  evaluators  who  evaluate  performance  risk 
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from  those  evaluators  who  evaluate  the  basic 
proposal  factors  of  cost , technical,  management, 
etc.  An  important  function  of  this  independent 
assessment  group  is  to  insure  that  only  relevant, 
recent  and  accurate  past  performance  information 
is  considered  in  the  evaluation. 

3.  Centralized  starting  point.  Evaluators  must  have 
available  to  them  a  central  system  that 
expeditiously  provides  them  sources  of  information 
other  than  these  cited  in  the  proposals.  These 
systems  of  data  collection,  storage  and  retrieval 
are  covered  in  section  F  of  this  chapter. 


3.  GAO 

GAO  will  give  great  deference  to  the  procurement 
agency's  perception  of  a  contractor's  past  performance 
provided  that  the  Government  follows  the  ground  rules  set 
forth  in  the  solicitation,  and  reasonably  considers  the 
contractor's  data  [Ref.  22:  p.  2].  The  Government  clearly  has 
the  right  to  consider  information  outside  of  an  offeror's 
proposal  to  evaluate  past  performance  [Ref.  22:  p.  2].  Past 
performance  is  a  responsibility-type  criteria,  and  it  may  be 
used  as  an  evaluation  criteria  in  source  selections  only  to 
make  a  graded  assessment  of  the  relative  merits  of  an 
individual  proposal,  or  to  determine  the  likelihood  that  a 
particular  offeror  will  successfully  perform  a  contract  based 
on  the  contractor's  record  under  other  contracts  (i.e.  a  PRA)  . 
If  past  performance  is  used  to  determine  the  acceptability  or 
unacceptability  of  a  proposal  on  a  "pass/fail"  basis,  an 
unacceptability  determination  will  likely  be  treated  by  GAO  as 
tantamount  to  a  non-responsibility  determination  [Ref.  22:  p. 
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3] .  As  is  the  case  in  making  non-responsibility 
determinations,  the  ground  rules  for  considering  performance 
data  in  the  source  selection  process  are  that  the  extrinsic 
data  must  be  current,  accurate  and  relevant  to  the  contract 
being  considered  for  award  [Ref.  22:  p.3  ]. 

The  Government  is  not  required  to  allow  offerors  to 
review  and  rebut  the  sources  or  references  which  the 
Government  utilizes  when  evaluating  past  performance.  However, 
the  Government  should  take  every  reasonable  measure  to  insure 
the  accuracy  of  the  data  relied  upon  during  the  PRA.  To  the 
extent  that  the  extrinsic  data  gives  rise  to  "deficiencies,  " 
the  Government  has  the  normal  duty  to  disclose  such 
deficiencies  along  with  any  others  found  during  the  evaluation 
process  to  those  offerors  within  the  competitive  range  for 
negotiations  [Ref.  22:  p.  2].  Also,  the  evaluation  of  a 
contractor's  past  performance  does  not  relieve  the  contractor 
of  its  burden  of  proving  the  technical  acceptability  of  its 
proposal.  Stated  differently,  the  Government  is  under  no 
obligation  to  seek  out  extrinsic  data  to  cure  deficiencies 
contained  in  a  proposal  [Ref.  22:  p.  3]. 

E.  CONDUCT  OF  A  PERFORMANCE  RISK  ASSESSMENT  (PRA) 

A  PRA  is  a  process  designed  to  determine  a  confidence 
measure  of  an  offeror's  potential  to  perform  the  requirements 
of  a  solicitation.  Generally,  it  includes  a  review  of  each 
offeror's  past  performance  and  an  assessment  of  the  current 
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risk  each  offeror  poses  based  on  how  they  have  performed  in 
the  past.  This  analysis  may  be  conducted  by  a  team  within  the 
SSEB,  but  separate  from  the  proposal  cost/technical  evaluation 
team  as  recommended  by  the  DoD/Joint  Service/Agency  PAT,  or  it 
may  be  conducted  by  a  group  totally  separate  from  the  SSEB, 
called  the  Performance  Risk  Assessment  Group  (PRAG)  as 
recommended  by  the  ACEC .  In  either  case,  the  results  of  the 
PRA  are  forwarded  to  either  the  SSAC,  the  SSA  or  the  PCO, 
depending  on  the  organization  of  the  procuring  agency.  The 
SSAC,  SSA  or  PCO  then  utilize  this  data  in  concert  with  the 
other  evaluation  results  co  determine  overall  factor  ratings 
and  competitive  ranges,  to  determine  weakness,  deficiencies 
and  questions  for  negotiations,  and,  finally,  to  select  the 
awardee  [Ref.  21:  P.  E-3  ]. 

The  SSEB  PRA  team  or  the  PRAG  should  consist  of  personnel 
with  technical  and/or  contractual  expertise,  as  well  as  skills 
in  the  following  areas  [Ref.  21:  p.  E-4]: 

1.  Ability  to  conduct  meaningful  interviews  in  an 
unregulated,  open-ended  environment. 

2.  Ability  to  assimilate  voluminous  amounts  of  data, 
arrive  at  appropriate  conclusions,  and  communicate 
those  conclusions  effectively  both  orally  and  in 
writing . 

3.  Ability  to  evaluate  an  offeror's  TQM  system  or  its 
equivalent  based  on  past  performance  data,  and 
identify  and  assess  the  principles,  practices, 
tools  and  techniques  of  continuous  process 
improvement . 
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It  is  also  highly  recommended  that  each  member  of  the  PRA  team 
receive  formal  training  in  area  3  above  through  an  in-depth 
TQM  training  course,  such  as  the  one  offered  at  the  Defense 
Systems  Management  College  (DSMC)  [Ref.  19:  p.  9]. 

The  necessary  data  and  sources  of  data  to  perform  the  PRA 
is  covered  in  more  de-tail  by  Sections  E  and  F  of  this  chapter. 
Once  the  PRA  team  j  s  assembled  and  trained,  the  information 
gathered  on  all  the  offerors  within  the  competitive  range  is 
assessed  through  a  five  step  process  to  develop  performance 
risk  ratings. 

STEP  1.  The  very  first  step  the  PRA  team  should  take  is  to 
establish  a  methodology  for  assessing  the  degree  of  risk  and 
to  identify  the  advantages  and  disadvantages  of  the 
performance  of  this  contract  by  a  given  offeror  based  upon  his 
past  performance. 

STEP  2.  Next,  each  member  should  become  very  familiar  with 
the  statement  of  work  and  specifications  in  the  RFP . 

STEP  3.  In  the  next  step,  they  should  sift  through  the 
collected  information  and  data  to  determine  the  relevancy  of 
present  and  past  contract  efforts  to  the  solicitation 
requirements.  It  is  important  for  the  PRA  team  to  consider,  at 
this  stage,  the  fact  that  information  pertaining  to  the 
proposing  prime  contractor  division  may  not  be  broad  enough  in 
scope  if  other  divisions,  affiliates,  subsidiaries,  critical 
subcontractors  or  teaming  contractors  also  perform  a  cr:'tical 
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element  of  the  contract,  or  significantly  influence  the 
proposed  effort  [Ref.  21:  p.  6]. 

PRA' s  should  consider  both  the  number  and  the  severity  of 
problems,  the  effectiveness  of  the  corrective  actions  taken, 
and  the  overall  work  record  of  the  offeror  and  the  other  key 
participants  [Ref.  21:  p.  6].  The  assessment  of  performance 
risk  is  not  intended  to  be  a  simple  arithmetic  function  of  an 
offeror's  performance  on  a  list  of  prior  contracts.  The 
information  deemed  most  relevant  and  significant  by  the  PRAG 
should  receive  the  greatest  consideration,  or  the  most  weight 
[Ref.  21:  p.  6].  The  degree  to  which  the  offeror  has 
demonstrated  tnat  past  and  present  problems  have  been  resolved 
or  that  planned  performance  has  been  achieved  will  be  an 
indicator  of  the  ability  of  the  contractor  to  successfully 
achieve  planned  performance  on  the  contract  [Ref.  19:  p.  8]. 
The  following  definitions  of  performance  risk  are  the  most 
commonly  used  and  most  highly  recommended  [Ref.  21:  p.  7]: 

1.  High  risk-  Significant  doubt  exists,  based  on  the 
offeror's  performance  record,  that  the  offeror  can 
perform  the  proposed  effort. 

2.  Moderate  risk-  Some  doubt  exists,  based  on  the 
offeror's  performance  record,  that  the  offeror  can 
perform  the  proposed  effort. 

3.  Low  risk-  Little  doubt  exists,  based  on  the 
offeror's  performance  record,  that  the  offeror  can 
perform  the  proposed  effort. 

4.  Unknown  (neutral)  risk-  No  performance  record 
identifiable . 
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A  basic  rule  of  thumb  for  the  assignment  of  the  above  risk 
levels  is;  if  an  offeror  has  performed  consistently  well  in 
the  past,  then  that  offeror  would  be  considered  a  low  risk 
with  regard  to  the  instant  procurement.  If  an  offeror's 
previous  performance  has  been  inconsistent,  then  patterns 
leading  to  that  spotty  performance  should  be  analyzed  and  a 
correspondingly  higher  risk  assessment  made.  An  offeror  who 
has  demonstrated  consistently  poor  performance  would  be 
considered  a  high  risk  [Ref.  21:  p.  E-10]. 

STEP  5.  In  the  event  that  performance  data  gives  rise  to 
a  "deficiency,"  the  PRA  group  should  provide  the  contracting 
officer  with  a  summary  of  derogatory  performance  information 
collected  for  each  offeror  for  discussions  [Ref.  22:  p.  17] 
As  another  means  for  insuring  the  accuracy  of  the  data 
collected,  the  contracting  officer, in  conducting  discussions, 
should  provide  the  offeror's  the  opportunity  to  comment  on 
these  deficiencies. 

STEP  6.  The  final  step  should  be  the  complete 
documentation  of  the  PRA  effort,  and  the  forwarding  of  these 
documents  to  the  appropriate  SSAC,  SSA  or  PCO.  The 
documentation  should  address  [Ref.  21:  p.  7]:  1.  the  sources 
of  performance  data,  2.  the  relevancy  and  significance  of  the 
data,  3.  the  currency  of  the  data,  4.  the  Performance  Risk 
Assessment  of  each  offeror,  and  5.  the  supporting  rationale 
for  each  performance  assessment.  It  is  important  to  note  that 
all  performance  data  gathered  by  or  included  in  documentation 


50 


used  by  the  PRA  team,  is  considered  as  source  selection 
information,  and  should  be  protected  against  unauthorized 
disclosure  [Ref.  21:  p.  7]. 

F.  PRIMARY  SOURCES  OF  PERFORMANCE  DATA  FOR  THE  PRA 

There  are  numerous  sources  of  contractor  present  or  past 
performance  information  and  data  for  the  PRA  teams  today. 
Some  sources  are  centralized  data  bases  for  a  service,  agency, 
of  DoD  wide  use,  and  are  known  as  primary  sources.  Some  are 
decentralized  into  various  local  contracting  activity  files  or 
DCAS  audits/on-site  visits,  and  some  are  telephone  interviews 
or  written  inquiries  from  knowledgeable  points  of  contact, 
known  as  secondary  sources.  Before  a  more  in-depth  review  is 
made  of  these  sources  of  information  and/or  data,  a 
clarification  of  the  difference  between  a  past  performance 
application  system  and  a  past  performance  data  base  must  be 
made  [Ref .  23]  : 

Contractor  Past  Performance  Application  System  (AS) —  any 

system  that  uses  contractor  past  performance  data  in  support 
of  award  decisions  for  a  specific  source  selection  or 
procurement . 

Contractor  Past  Performance  Data  Base  System  (DBAS) —  any 

system  for  the  primary  purpose  of  the  collection  of 
contractor  past  performance  data  for  general  use. 

The  following  is  the  most  current  inventory  of  DoD 
contractor  past  performance  data  collection  initiatives  by 
Service  or  Agency,  with  a  coding  as  to  whether  they  are  a 
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contractor  past  performance  data  base  system  or  a  mixture  of 
both  a  data  base  system  and  an  application  system  [Ref.  23]: 

NAVY 


1 .  Navy  Product  Deficiency  Reporting  and  Evaluation 
System :  (DBAS)  An  automated  Navy  central  data  base 
system  for  recording  information  on  the  historic 
quality  performance  of  Navy  contractors.  This 
system  is  supported  by  a  group  of  internal  (i.e. 
Contractor  Evaluation  System,  Quality  Deficiency 
Report  System,  Report  of  Discrepancy  System)  and 
external  (i.e.  DLA  Alert  List,  DLA  Method  C,D,E 
Program)  feeder  data  bases,  and  is  managed  by 
NMQAO  in  Portsmouth,  New  Hampshire. 


ARMY 


2 .  Army  Contractor  Information  System  (CIS)  : 

(DBAS/AS)  Contains  both  a  data  system  (Contractor 
Information  Reports)  and  a  corresponding 

application  methodology.  A  CIR  contains 

administrative  data,  a  "network"  of  points  of 
contact,  and  alerts  to  quality,  delivery,  and  cost 
performance  problems.  CIR' s  are  used  during 
source  selection  to  supplement  contractor-supplied 
information  and  data  from  other  government  sources 
to  support  a  performance  risk  analysis. 

3 .  Army  Contractors  Requiring  Special  Attention 
(CRSA) :  (DBAS)  A  data  base  implemented  by  the  Army 
Materiel  Command  which  identifies  and  gives 
special  attention  to  a  contractor  experiencing 
problems  in  delivery  delinquencies,  quality  and/or 
cost.  Contractors  are  notified  when  placed  on  the 
program  and  are  tasked  with  developing  "get-well" 
plans . 

4 .  Army  Commodity  Command  Standard  System  (CCSS) : 
(DBAS)  This  is  the  Army's  wholesale  logistics  data 
base  management  system  which  operates  at  the  Army 
Materiel  Command's  six  commodity  commands.  It 
links  financial  management,  material  management, 
and  contracting  information  into  unified  data 
bases.  Available  data  relevant  to  performance 
include  contractor  delivery  and  award  history. 
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5. 


Army  Contractor  Performance  Certification  Program 
(CP/2) :  (DBAS)  An  automated  data  base  that  tracks 
contractors  who  are  voluntarily  enrolled  in  (CP/2) 
and/or  certified  for  good  quality  and  production 
performance.  Certified  status  is  granted  after  the 
contractor  successfully  achieves  milestones  which 
show  improvement  in  process  control,  manufacturing 
discipline  and  product  quality.  Certified  status 
is  made  available  to  contracting  officers  for 
consideration  in  source  selection  decisions. 

6.  Army  Deficiency  Reporting  System  (DRS) :  (DBAS) 
This  is  an  automated  data  base  that  track  customer 
complaints  on  material  in  the  supply  system  or  the 
field.  Customer  complaints  are  thoroughly 
investigated,  contractor  liability  determined  and 
corrective  action  initiated.  When  contractor 
liability  is  determined  on  an  excessive  number  of 
customer  complaints  against  a  specific  item,  then 
Quality  Assurance  informs  Procurement  that  the 
manufacturer  is  a  poor  risk  for  future  awards. 

7.  Army  Contractor  History  Files:  (DBAS)  Data  bases 
developed  separately  by  individual  contracting 
activities  in  the  Army  Materiel  Command  to  record 
quality  history  on  individual  contracts.  Data  can 
be  manipulated  to  synopsize  quality  performance  of 
specific  contractors.  Information  forwarded  to 
contracting  officers  on  request. 


AIR  FORCE 


8 .  Air  Force  Systems  Command  Contractor  Performance 
Assessment  Report  (CPAR) :  (DBAS)  This  manual  data 
base  system  proyides  detailed  information  and  an 
assessment  of  the  on-going  performance  of  selected 
contractors  on  major  systems  procurements.  The 
data  base  consists  of  the  narrative  assessment  of 
the  project  manager  as  supported  by  his  functional 
experts,  coupled  with  surveys  and  questionnaires 
returned  by  other  agencies.  Each  buying  division 
has  a  library  of  all  CPAR' s .  The  data  is  available 
to  officials  for  consideration  in  making  new 
contract  awards. 

9  .  Air _ Force _ Logistics _ Command  Contractor 

Responsibility  Review  Program  (CRRP) :  (DBAS /AS) 
This  automated  data  base  and  application  system  is 
used  to  support  responsibility  determinations  for 


53 


smaller  procurements.  Inputs  include  quality  and 
delivery  data. 


DEFENSE  LOGISTICS  AGENCY 


1 0 .  Defense  Logistics  Agency  (DLA)  Contractor  Profile 
System:  (DBAS)  This  proposed  data  base  system  will 
contain  various  historical  views  of  contractor 
information  from  preaward  survey  data  through 
delivery  of  the  end  item.  This  system  is  broad 
based  and  will  contain  data  on  production, 
quality,  contract  administration,  financial 
services,  engineering,  property,  procurement,  and 
legal.  This  system  is  intended  to  be  an 
information  sharing  system  with  the  military 
services  tied  in  via  the  DoD  network. 

1 1 .  DLA  Standard  Automated  Materiel  Management  System 

(SAMMS) :  (DBAS)  This  is  the  DLA  interactive, 

multifunctional  data  base  system.  It  is  comprised 
of  the  requirements,  distribution,  technical, 
financial  and  contracting  subsystems.  Among  its 
other  functions,  the  latter  component 
automatically  generates  purchase  requests  and 
trailers  containing  past  performance  data. 

12 .  DLA  Mechanization  of  Contract  Administration 
Services  (MOCAS) :  (DBAS)  This  is  the  present  data 
base  system  concerning  contracts  assigned  to  DLA 
for  administration.  This  system  contains  much 
current  delivery  history,  financial  information, 
and  other  contract/  contractor  data.  Some  of  the 
military  services  are  already  tied  in  to  this 
system . 

13.  DLA  Preaward  Survey  System  (PASS) :  (DBAS)  This 
automated  data  base  system,  designed  as  a  subset 
to  an  overall  contractor  profile  data  base  system, 
provides  information  related  to  a  contractor's 
past  performance.  The  PASS  includes,  but  is  not 
limited  to,  production,  quality  assurance, 
transportation,  property  and  financial/accounting 
information  arranged  in  a  series  of  on-line 
preaward  survey  forms.  DLA  provides  access  to 
buying  activities  for  on-line  queries. 

14.  DLA  Customer/Depot  Complaint  System  (CDCS) :  (DBAS) 
This  is  a  standardized  data  base  system  designed 
to  track  and  monitor  quality  complaints  received 
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by  the  individual  supply  centers  from  customers, 
supporting  supply  activities,  and  service  and  DLA 
depots.  Types  of  complaints  received  include 
QDR's,  rod's  and  storage  quality  control  reports. 
CDCS  interfaces  with  SAMMS . 

15.  DLA  Quality  Evaluation  Program;  (DBAS)  The  QEP  is 
a  quality  history  data  base  system  developed  in 
response  to  IG  and  GAO  findings  that  DLA  was  not 
using  available  Government  quality  data  in 
awarding  contracts.  QEP  is  currently  used 
specifically  to  aid  in  an  assessment  of  a 
contractor  responsibility.  Data  is  maintained  on 
both  a  contractor  and  an  item  basis.  QEP  includes 
information  on  first  article  testing,  postaward 
orientation  conferences,  quality  systems 
management  visits  and  reviews,  product  waivers  and 
deviations,  quality  assurance  letters  of 
instruction,  contracts  reviewed  by  the  quality 
element,  and  special  quality  assurance  action 
information . 


These  are  the  primary  sources  of  information  for  each 
respective  service  or  agency,  and  there  is  nothing  which 
prevents  one  service  from  using  the  data  base  of  another 
service  or  DLA.  None  of  the  data  bases  discussed  in  the 
preceding  paragraph  were  specifically  designed  for  use  with  a 
particular  methodology  for  conducting  past  performance 
evaluations  in  source  selections.  They  all  lacked  consistency 
in  data  elements,  and  because  they  are  maintained  for 
different  purposes,  there  is  a  lack  of  uniformity  in  how  the 
data  is  formatted  [Ref.  22:  p.  9].  Further,  none  of  the  data 
bases  include  performance  information  for  all  types  of 
contracts  and  requirements. 
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6.  SECONDARY  SOURCES  OF  PERFORMANCE  DATA  FOR  THE  PRA 

There  are  many  secondary  sources  of  data  to  supplement 
these  primary  sources,  and  these  include,  but  are  not  limited 
to  the  following  [Ref.  24]: 


1.  Certificates  of  Competencies  (COC^s):  If  a  small 
business  is  determined  to  be  non-responsible  by  a 
Government  buying  activity,  the  small  business  can 
request  that  the  Small  Business  administration 
determine  whether  the  business  is  responsible.  If 
the  SBA  determines  that  the  small  business  is 
responsible,  then  the  SBA  will  prepare  a  COC  to 
document  that  determination,  and  will  send  a  copy 
to  NMQAO  for  inclusion  in  PDREP . 

2.  Quality  System  Reviews:  Performed  by  DCMC,  they 
involve  an  evaluation  of  the  contractor' s  quality 
procedures  and  verification  that  the  contractor's 
quality  practices  conform  to  those  procedures. 
Navy  activities  receive  copies  of  quality  system 
reviews  if  they  request  a  copy. 

3.  Product  Orienced  Surveys:  These  are  technical 
product  inspections  conducted  in  a  contractor' s 
plant  when  a  buying  activity  desires  to  perform  a 
special  test  on  an  item.  They  are  performed  by 
DCMC  when  requested  by  the  buying  activity.  Copies 
of  them  are  available  upon  request  from  NMQAO. 

4.  Discussions  with  technicians  and  program  managers 
from  actual  performance  experience,  where 
available . 

5.  Contractor  History  Files:  Data  bases,  both  manual 
and  automated,  maintained  separately  by  individual 
DoD  contracting  activities  to  record  quality 
history  on  individual  contracts. 

6.  Visits  to  contractor  facilities  by  PRA  team 
members. 

7.  Receipt  Inspection  Records 

8.  Production  Lot  Testing  reports 

9.  References  cited  by  offerors  in  their  proposals. 
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10.  Telephone  interviews  or  written  inquiries  with 
selected  Points  of  Contact  (POCs)  in  any  way 
knowledgeable  of  a  contractor's  previous  efforts. 

H.  SUMMARY 

This  chapter  examined  performance  risk  from  a  legal 
standpoint,  described  a  five  step  procedure  for  performing  a 
PRA,  and  listed  and  described  the  various  primary  and 
secondary  sources  of  past  performance  data  available  to  DoD 
contracting  activities  today.  The  next  chapter  will  present 
the  methodology  used  to  obtain  feedback  on  the  source 
selection  procedure  model  from  DoD  personnel,  and  will 
summarize  some  of  the  feedback  received  and  modifications  made 
to  the  model  in  response  to  the  feedback. 
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V.  FEEDBACK  AMD  ANALYSIS 


A.  INTRODUCTION 

This  chapter  will  present  the  methodology  used  to  obtain 
feedback  on  the  source  selection  model.  The  feedback  was 
obtained  from  knowledgeable  and  experienced  PCO's,  contracting 
policy  officials,  contract  specialists  and  instructors  in 
contracting,  and  the  DoD/ Joint  Service/Agency  PAT.  The 
information  received  will  be  presented  by  activity  submitting 
it  and  by  classification  of  the  response.  The  comments  not 
supporting  the  model  will  be  analyzed  and  addressed 
separately.  Modifications  to  the  model  based  on  this  feedback 
will  also  be  described. 

B .  METHODOLOGY 

The  refinement  of  the  source  selection  model  presented  in 
Chapter  III  and  Appendices  C,  D  and  E  should  be  an  iterative 
process  utilizing  revisions  based  on  input  from  DoD 
contracting  activities  as  well  as  commercial  purchasing 
departments,  and  by  use  of  an  actual  pilot  test  at  a  field 
contracting  activity.  The  first  step  in  this  refinement 
process  consisted  of  the  forwarding  of  Chapter  III  and 
Appendices  C,  D  and  E  along  with  a  cover  memorandum  to 
knowledgeable  and  experienced  naval  officers  and  senior  civil 
service  officials  working  in  the  contracting  or  policy 
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divisions,  or  in  teaching/instruction  billets  of  six 
significant  Navy  commands:  Naval  Air  Systems  Command  (NAVAIR)  , 
Naval  Sea  Systems  Command  (NAVSEA)  ,  Military  Sealift  Command- 
Pacific  (MSCP) ,  Naval  Supply  Center-  Puget  Sound,  the  Office 
of  the  Assistant  Secretary  of  the  Navy  (Research,  Development 
and  Acquisition)  ,  and  the  Naval  Postgraduate  School,  Monterey 
(NPS)  .  In  addition,  a  copy  was  forwarded  to  the  Navy 
representative  on  the  DoD/ Joint  Service/Agency  PAT,  Mr. 
Richard  Findley. 

The  recipients  of  the  source  selection  model  were 
requested  to  read  the  model  and  provide  constructive  feedback 
on  any  aspect  of  the  model  in  which  they  may  have  knowledge 
and  experience.  It  was  made  clear  that  all  comments  regarding 
technical,  theoretical,  procedural,  legal  or  practical  aspects 
were  encouraged  for  submission.  All  recipients  with  the 
exception  of  NSC  Puget  Sound  responded  to  the  feedback 
request . 

C.  FEEDBACK  AND  ANALYSIS 

The  responses  to  the  feedback  request  will  be  summarized 
by  command  or  group  under  four  different  headings  based  upon 
the  category  they  fell  within:  comments  in  favor  of  the 
general  concept,  comments  not  in  favor  of  the  general  concept, 
comments  in  favor  of  specific  aspects  of  the  source  selection 
model,  and  comments  not  in  favor  of  specific  aspects  of  the 
source  selection  model.  If  a  command  did  not  have  a  response 
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in  one  or  more  of  the  four  categories,  it  will  not  appear 
under  that  category.  The  comments  that  weren't  in  favor  of  the 
general  concept  and  specific  aspects  of  the  source  selection 
model  will  be  addressed  as  a  group  after  subsections  (2)  and 
(4)  . 


1 .  Comments  in  Favor  of  the  General  Concept 

NAVAIR  [Ref.  25] : 

The  incorporation  of  TQM  into  the  proposal  evaluation 
process  is  an  important  and  worthy  goal. 

The  model  provides  an  excellent  template  for  others 
interested  in  incorporating  TQM  with  the  evaluation 
process.  I  think  that  the  model  achieves  many  of  its 
objectives . 

The  model  offers  a  good  starting  point,  but  it  must  be 
altered  to  fit  the  needs  and  circumstances  of  the  individual 
procurement . 

NAVSEA  [Ref.  26]: 

TQM/TQL  in  the  source  selection  process  is  workable,  and  the 
concept  is  very  current  and  useful.  The  effort  put  forth  in 
constructing  this  model  is  noteworthy. 

ASN(RD&A)  [Ref.  28]: 

I  believe  the  technique  you  have  developed  along  with  the 
rating  categories  are  very  good. 

NFS  [Ref.  29] : 

I  think  this  is  a  great  concept. 

Good  ideas  and  lots  of  work/thought  evident. 

2.  Comments  That  Weren't  in  Favor  of  the  General  Concept 
NAVAIR  [Ref.  25]: 

In  general,  parts  of  the  model  are  OK-  most  parts  are  too 
cumbersome  and  more  trouble  than  they  are  worth. 
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NAVSEA  [Ref.  26]: 


If  the  procurement  process  utilizing  this  model,  or  some 
version  of  it,  is  not  performed  correctly,  contractors  may 
try  to  resort  to  brochuremanship  to  make  the  Government  ■ 
happy,  the  last  thing  the  Government  wants. 

ASN(RD&A)  [Ref.  28]: 

My  concern  is  not  with  the  approach  you  are  taking,  but  in 
my  professional  view,  the  "over  taxing"  of  our  source 
seleccion  process.  Application  of  TQM  into  complex  source 
selections  which  already  require  detailed  analysis  and 
review  by  evaluators  is  not  sound. 

To  believe  that  we  would  pick  a  contractor  based  on  his  TQM 
package  on  a  major  source  selection  is  extremely  short 
sighted . 

My  real  concern  is  we  are  compounding  the  source  selection 
process  with  too  many  items  which  are  effectively  driving  us 
more  and  more  away  from  fully  using  our  "common  sense" 
capabilities  when  conducting  a  source  selection.  The  fact 
that  GAO  and/or  a  court  upheld  our  actions  doesn't  mean  it 
is  good  business.  In  source  selection,  simple  is  always 
better . 

I  regret  that  I  cannot  be  more  receptive  to  your  paper  but 
my  concerns  are  "good  idea"  but  by  the  "wrong"  approach 

Source  selection  evaluations  should  not  be  the  panacea  to 
ensure  that  TQM  is  embraced. 

NPS  [Ref.  29]: 

Most  of  the  subfactors  in  Appendix  C  are  too  detailed  and 
administratively  burdensome. 

I  quickly  came  to  the  conclusion  that  I  want  to  be  the 
incumbent  contractor  when  this  SSEB  meets.  If  company  "A" 
has  performed  this  contract  previously  and  has  data  or 
"evidence"  to  support  its  contentions  regarding  TQM/SPC 
records,  then  I  see  it  as  having  a  large  comparative 
advantage  over  competitors. 

DoD/ Joint  Service /Agency  PAT  [Ref.  30]: 

Your  approach  of  developing  evaluation  criteria,  while 
technically  acceptable,  may  not  be  viewed  favorably  by 
contractors  . 
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The  central  theme  of  the  comments  not  in  favor  of  the 


general  concept  was  that  the  process  of  incorporating  TQM  into 
the  source  selection  process  was  too  administratively 
burdensome,  and  that  it  complicated  an  already  complex  system 
to  the  point  where  common  sense  could  no  longer  be  effectively 
applied.  Other  points  reflected  the  thought  that  the  process 
would  result  in  brochuremanship  from  contractors,  and  that  the 
incumbent  contractors  would  have  an  inherent  advantage  over 
other  competing  firms. 

Chapter  III  pointed  out  that  this  model  is  designed  to 
be  flexible,  and  part  of  its  flexibility  is  that  it  need  not 
be  utilized  in  its  entirety.  A  contracting  activity  may  use 
those  subfactors  or  subordinate  subfactors  which  are 
applicable  to  the  particular  procure.ment .  In  order  to  make 
the  model  more  streamlined  and  less  administratively 
burdensome  without  compromising  the  intent  of  the  procedure, 
nine  of  tne  original  23  subordinate  subfactors  were  deleted 
because  they  overlapped  other  subordinate  subfactors,  or  were 
either  too  difficult  to  evaluate  or  were  inapplicable  to  DoD 
acquisition . 

The  other  two  points  may  not  be  potential  sources  of 
problems  or  complications  if  the  model  process  is  conducted 
properly.  The  PRA  is  designed  to  evaluate  actual  evidence  of 
past  and  current  performance  independent  from  the  offeror's 
proposal.  If  the  offeror  resorts  to  "brochuremanship"  or 
"fluff,"  the  PRA  will  detect  the  anomaly,  and  will  reflect  the 
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offeror's  true  performance  in  the  overall  rating.  Incumbents 
should  not  have  an  inherent  advantage  if  the  contracting 
activity  tailors  the  source  selection  process  to  minimize 
bias.  If  firms  other  than  the  incumbent  do  not  have 
significant  past  experience  which  can  be  evaluated  by  the  PRA, 
then  the  SSEB  can  assign  a  "neutral"  risk  to  the  offeror  on 
the  PRA.  A  "neutral"  risk  level  will  not  adversely  affect  the 
overall  factor  rating.  If  the  incumbent's  performance  has  been 
either  "outstanding"  or  "marginal"  ,  then  this  fact  will  be 
appropriately  reflected  in  the  evaluations. 

This  model  was  not  designed  with  the  intent  of  placing 
TQM  or  other  quality  factors  as  the  sole  or  even  as  the  major 
source  selection  criteria  for  major  systems  acquisitions.  TQM 
is  but  one  subfactor  among  many  factors  and  subfactors.  This 
is  why  the  model  designed  TQM  to  be  subfactors  and  subordinate 
subfactors,  as  they  are  a  part  of  a  more  encompassing  set  of 
evaluation  criteria.  TQM  can  play  as  large  or  as  small  a  role 
in  source  selection  decisions  as  the  contracting  activity 
desires.  This  is  the  flexibility  built  into  this  model. 

3 .  Comments  in  Favor  of  Specific  Aspects  of  the  Model 

NAVAIR  [Ref.  25]: 

The  inclusion  of  a  requirement  for  TQM  training  for 
evaluators  is  excellent.  This  is  essential  to  an  effective 
evaluation . 

MSCP  [Ref.  27]: 

I  like  the  concept  of  strong  and  weak  point  narratives  in 
Appendix  C. 
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DoD/ Joint  Service /Agency  [Ref.  30] : 

Your  expansion  on  "Instructions  to  Offerors"  is  good... and 
provide  good  overall  information  that  goes  a  long  way  toward 
ensuring  that  the  offerors  don't  attempt  to  provide  "fluff" 
instead  of  substantiated  data. 

4.  Comments  That  Weren't  in  Favor  of  Specific  Aspects  of 
the  Model 

These  comments  were  not  taken  for  action  in  the 
modification  of  the  model. 

NAVAIR  [Ref.  25] : 

An  important  concern  related  to  the  model  is  the  reliance  on 
numerical  scoring...!  would  require  an  adjective  rating  for 
each  listed  criteria,  preferably  with  comments/rationale 
from  the  evaluators  to  support  the  adjectives  assigned. 

I  think  that  23  pages  of  evaluation  forms  is  excessive  and 
is  likely  to  become  counter-productive  as  evaluators 
struggle  with  that  much  paperwork.  It  also  encourages 
proposals  to  be  excessively  lengthy. 

NFS  [Ref.  29]: 

Appendix  D  looks  OK-  but  if  everything  in  Appendix  C 
translates  to  Appendix  D  and  evaluation  forms,  it's  too 
detailed  and  burdensome. 

DoD/Joint  Service /Agency  [Ref. 30]: 

Be  aware  that  Navy  policy  has  recently  been  emphasized  that 
numerical  ratings  are  not  to  be  used. 


The  central  theme  of  the  comments  that  weren't  in 
favor  of  specific  aspects  of  the  model  was  that  numerical 
scoring  techniques  were  discouraged  by  the  Navy,  and  makes  the 
acquisition  process  vulnerable  to  protests.  The  other  comments 
reflected  that  certain  areas  of  the  model  are  cumbersome. 
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The  Navy  guidance  was  based  on  a  policy  memorandum 

from  the  Assistant  Secretary  of  the  Navy  (Shipbuilding  and 

Logistics)  that  prohibited  the  assignment  of  a  numeric  score 

to  the  proposed  cost  or  price  in  a  best  value  decision  [Ref. 

31]  .  The  problems  this  memorandum  addresses  are  situations 

where  contracting  activities  assigned  numeric  scores  to  both 

cost/price  and  technical  evaluation  factors  to  arrive  at  a 

total  point  score.  The  source  selection  process  in  the 

researcher's  model  does  not  address  cost  or  technical  factors 

other  than  TQM  and  quality  in  general.  The  model  is  flexible 

enough  to  allow  for  numerical  scoring,  adjectival  ratings  or 

narrative  comments,  or  some  combination  thereof.  The  model 

used  a  combination  of  all  three  systems  to  illustrate  the 

flexibility  of  the  procedures.  In  addition,  the  policy 

memorandum  preserved  the  right  for  Navy  contracting  activities 

to  use  numerical  scoring  on  non-cost /price  factors  [Ref.  31] : 

Contracting  officers  may  continue  to  use  numeric  scoring 
in  evaluating  the  technical  portion  of  an  offeror's 
proposal . 

D.  MODIFICATIONS  TO  MODEL  BASED  ON  FEEDBACK 

The  following  modifications  have  been  made  to  the  source 
selection  model  based  upon  feedback  from  the  respondents: 

(1)  Every  subordinate  subfactor  contained  the  phrases 
"provide  credit  for"  and  "evidence  of."  These  phrases  were 
replaced  by  "examples  may  include  but  are  not  limited  to"  and 
"documentation/  facts  verifying  the  existence  of" 
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respectively,  because  some  respondents  pointed  out  that  the 
evaluation  criteria  as  written  made  the  Government  appear  as 
though  it  were  mandating  specific  TQM  practices  to 
contractors.  The  word  "evidence"  was  not  forceful  enough 
according  to  one  respondent,  who  felt  that  the  term  "evidence" 
did  not  adequately  describe  what  the  evaluators  needed  to  look 
for  when  evaluating  proposals. 

(2)  Under  subordinate  subfactor  A(l),  Senior  Executive 
Leadership,  the  example  of  "the  request  for  documentation  of 
senior  executive  leadership  in  and  communication  of  quality 
excellence  outside  the  company  to  various  businesses, 
professional  and  community  groups"  was  deleted  because  several 
respondents  felt  as  though  it  was  not  directly  applicable  to 
contracting  with  DoD,  and  that  it  invited  submission  of 
"fluff. " 

(3)  The  "long  term"  planning  mentioned  in  several 
subordinate  subfactors  were  changed  from  "3-5  years"  to  "3-5 
years  or  more"  because  a  respondent  made  the  case  that  3-5 
years  was  not  a  long  enough  time  frame  to  be  considered  as 
true  long  term  planning. 

(4)  Subordinate  subfactor  C(2),  Quality  Leadership 
Indicators  in  Planning,  was  deleted  because  several 
respondents  pointed  out  that  it  was  difficult  to  verify  and 
had  no  real  applicability  to  DoD  contracting. 

(5)  Several  respondents  felt  that  the  subordinate 
subfactor.  Human  Resource  Management,  of  the  Human  Resource 
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utilization  subfactor  was  already  covered  by  subordinate 
subfactors  D(l)  through  D(4).  Therefore,  the  Human  Resource 
Management  subordinate  subfactor  was  deleted  to  avoid  the 
double-weighting  of  this  area. 

(6)  The  subordinate  subfactors  "Quality  Assessment"  and 
"Documentation"  of  the  Quality  Assurance  of  Products  and 
Services  factor  were  deleted  entirely  because  there  was 
significant  overlap  between  it  and  the  subordinate  subfactors 
E(l)  through  EM)  . 

(7)  The  subordinate  subfactor  "Complaint  Resolution  for 
Quality  Improvement"  of  the  "Customer  Satisfaction"  subfactor 
was  deleted  entirely  because  there  was  significant  overlap 
between  it  and  the  subordinate  subfactors  F(l)  and  F{2), 

(8)  The  entire  "Past  Performance"  subfactor,  including  the 
subordinate  subfactors:  Quality  of  Products/Services 
Delivered,  Compliance  with  Schedule,  Integrated  Logistics 
Support  and  Compliance  with  Cost/Price  Parameters,  was  deleted 
because  of  many  respondents  pointing  out  the  significant 
potential  for  overlap  and  consequent  "double-weighting" 
possibility  for  the  past  performance  areas.  In  addition,  the 
PRA  evaluates  past  performance  almost  exclusively,  so  the 
potential  "double-weighting"  of  past  performance  between  the 
basic  proposal  evaluation  and  the  PRA  was  eliminated. 
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E.  SUMMARY 


This  chapter  presented  the  methodology  used  to  obtain 
feedback  on  the  source  selection  model  from  knowledgeable  and 
experienced  DoD  personnel,  and  summarized  some  of  the  feedback 
received  and  modifications  made  to  the  model  in  response  to 
the  feedback.  The  final  chapter  will  present  the  conclusions 
and  recommendations  of  the  thesis,  the  answers  to  the  research 
questions,  and  recommended  areas  for  further  research. 
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VI.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  INTRODUCTION 

This  chapter  will  discuss  the  conclusions  and 
recommendations  resulting  from  the  rhesis  research.  After 
presenting  the  conclusions  and  recommendations,  the  research 
questions  will  be  answered.  Finally,  recommendations  will  be 
made  concerning  areas  for  further  research. 

B.  CONCLUSIONS 

The  research  upon  which  this  thesis  is  based  consisted  of 
an  extensive  literature  review,  interviews  with  Government 
contracting,  legal  and  policy  personnel,  and  feedback  on  the 
source  selection  model  from  knowledgeable  and  experienced 
personnel  at  six  Navy  commands  and  the  DoD/Joint 
Service/Agency  Process  Action  Team.  Based  upon  this  research, 
the  researcher  concludes: 


1.  There  exists  in  DoD  today  many  programs, 
regulations,  maiidates  and  other  incentives  to 
incorporate  TQM  or  other  quality  factors  into  the 
source  selection  process,  just  as  commercial 
procurement  practices  do. 

2.  Navy  PCO's  are  willing  to  utilize  quality  factors 
in  their  source  selection  process  if  they  could 
have  access  to  a  feasible,  legal  and  generally 
accepted  method  of  source  selection  incorporating 
prestated  quality  measures,  such  as  TQM. 

3.  The  1990  ACEC  production  contract  for  third 
generation  night  vision  devices  was  awarded 
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utilizing  TQM  and  SPC  in  the  source  selection 
process.  ACEC' s  source  selection  process  was 
feasible  and  legal,  and  only  time  will  tell  if  the 
source  selection  process  was  successful  in  meeting 
the  Army's  goals  for  quality,  timeliness,  cost 
control  and  low  costs  of  ownership. 

4.  The  DoD/Joint  Service/Agency  PAT's  Draft  Guide  for 
Incorporating  TQM  in  Source  Selection  was  not 
finalized,  and  consequently  did  not  promulgate  a 
specific  or  general  model  source  selection  plan 
with  evaluation  guidelines  or  criteria. 

5.  The  model  source  selection  plan  described  in 
Chapters  III  and  IV  and  Appendices  C,  D  and  E  of 
this  thesis  is  a  synthesis  of  the  most  feasible 
and  viable  procedures  of  the  ACEC  procurement,  the 
PAT  draft  guidance  and  ideas  of  the  researcher. 

6.  The  model  source  selection  procedures  are  flexible 
enough  to  be  tailored,  expanded  or  otherwise 
modified  by  local  procurement  activities  into  a 
workable  source  selection  method. 

7.  Based  upon  an  unscientific  sampling  of  comments 
from  Navy  contracting  activities,  the  general 
climate  towards  acceptance  or  usage  of  the  source 
selection  model  is  favorable.  This  favorable 
acceptance  is  contingent  upon  the  requirement  that 
the  model  be  streamlined  and  less  burdensome  to 
both  Government  and  contractor,  and  that  the  local 
procuring  activity  be  able  to  tailor  or  modify  it 
as  necessary. 

8.  There  exist  pockets  of  strong  resistance  to  the 
incorporation  of  TQM  or  other  prestated  quality 
factors  into  the  source  selection  process  due  to  a 
variety  of  philosophies,  attitudes  and 
perceptions.  More  universal  acceptance  of  the 
model  or  the  concepts  behind  the  model  may  be 
gained  as  personnel  in  influential  policy  or 
contracting  positions  become  more  educated  on  or 
knowledgeable  of  TQM  or  other  quality  concepts. 

9.  The  construction  of  a  solid,  viable  and  feasible 

source  selection  model  which  incorporates  TQM  or 
other  quality  factors  into  the  DoD  source 
selection  process  is  an  iterative  process 

requiring  continuous  refinement,  feedback, 

modifications  and  pilot  testing. 
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C .  RECOMMENDATIONS 


Based  upon  the  research  and  conclusions,  this  researcher 
suggests  the  following  recommendations. 

1 .  The  DoD/ Joint  Service/Agency  PAT  should  continue 
efforts  to  complete  and  finalize  for  distribution,  the 
guidance  for  integrating  TQM  in  the  DoD  source  selection 
process.  If  or  when  this  is  done,  the  contents  of  Chapter 
III  and  Appendix  C  of  this  thesis  should  be  considered  for 
incorporation  into  the  final  draft. 

The  successor  to  the  Honorable  John  A.  Betti  will  have  an 
extensive  degree  of  influence  over  the  decision  whether  to 
continue  the  effort  to  finalize  the  Draft  Guide  or  not.  If 
the  decision  is  to  continue  with  the  guidance  effort,  then  the 
PAT  should  consider  the  contents  of  Chapter  III  and  Appendix 
C  of  this  thesis.  This  thesis  presents  logical  extensions, 
amplifications  or  modifications  of  the  Draft  Guide  based  on 
the  literature  research,  interviews  and  constructive  feedback 
from  Navy  contracting  activities. 

2 .  Efforts  should  be  made  by  future  researchers  to 
further  refine  this  researcher' s  source  selection 
procedures  model . 

As  recommended  in  Section  D  of  this  chapter,  this 
researcher' s  source  selection  procedures  model  requires 
further  refinement  and  update  based  upon  input  from  other 
Government  contracting  activities,  and  from  the  private  sector 
procurement  personnel.  An  excellent  method  to  further  refine 
it  would  entail  an  actual  pilot  procurement  test  at  an 
activity  such  as  NAC  Indianapolis  or  NWC  China  Lake. 
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3 .  Goverzuaent  contracting  activities  and  commercial  firms 
should  institute  or  expand  their  training  and  education 
programs  to  include  sessions  or  courses  on  the 
relationship  between  TQM  or  other  quality  factors  and  the 
source  selection  process. 

One  of  the  major  impediments  to  acceptance  or  utilization 
of  source  selection  procedures  which  incorporate  TQM  is 
education  and  awareness  for  both  Government  and  commercial 
personnel.  Inflexible  attitudes  towards  quality  factors  and 
TQM  are  due  in  large  part  to  fears  of  the  unknown, 
misconceptions  and  resistance  to  change.  Such  barriers  can  be 
removed  or  lessened  most  effectively  through  unbiased, 
objective  presentation  of  the  basic  fundamentals  and 
definitions  of  quality  and  TQM  as  they  relate  to  DoD 
contracts . 

4 .  The  Navy  should  make  an  attempt  to  bridge  the 
significant  gap  between  it  and  the  other  services  and  DIA 
in  developing  and  utilizing  quality  history  data  bases 
which  can  incorporate  contractor  past  performance  into  the 
source  selection  process . 

In  terms  of  the  quantity,  quality  and  maturity  of  data 
management  systems  which  collect,  store  and  disseminate 
contractor  quality  history  for  contractor  responsibility  or 
performance  risk  determinations,  the  Navy  is  far  behind  the 
other  services  and  DLA.  This  point  is  reinforced  graphically 
by  a  cursory  review  of  the  answer  to  subsidiary  question 
number  5  in  section  D  of  this  chapter. 
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O.  ANSWERS  TO  THE  RESEARCH  QUESTIONS 

Primary  Research  Question:  What  would  be  a  useful  and 
practical  source  selection  procedure  model  for  PCO' s  and 
SSEB' s  which  would  incorporate  significant  and 
quantifiable  quality  factors  with  a  procedure  to  verify 
and  validate  offered  contractor  quality  information  with 
contractor  quality  history? 

A  useful  and  practical  source  selection  evaluation 
procedure  model  would  be  a  synthesis  of  the  ACEC' s  method  of 
procurement  for  the  third  generation  night  vision  devices, 
with  the  evaluation  guidelines  or  criteria  suggested  by  the 
DoD/ Joint  Service/Agency  PAT  in  their  Draft  Guide.  The  model 
in  its  most  basic  form  consists  of  the  determination  of  an 
adjective  rating  and  a  performance  risk  level  for  each 
evaluation  subordinate  subfactor  based  upon  an  evaluation  of 
the  basic  proposal  and  a  Performance  Risk  Assessment 
respectively.  These  separate  evaluations  would  be  conducted 


by  two  independent  teams 

within 

the  SSEB 

,  and 

the  basic 

proposal  rating  and 

the 

risk  assessment 

level 

would  be 

combined  by  the  SSEB 

to 

form  an 

overall 

rating 

for  each 

evaluation  factor. 

The 

overall 

ratings 

plus 

supporting 

documentation  and  narrative  comments  would  be  forwarded  by  the 
SSEB  to  the  SSAC  or  SSA  for  a  final  award  decision. 

This  model  procedure  is  flexible  because  numerical 
weighting,  adjectival  descriptions  and  narrative  comments  can 
all  be  used  either  alone  or  in  some  combination  with  the 
others.  The  evaluation  criteria  for  the  basic  proposal  are 
modified  application  preparation  guidelines  for  the  Malcolm 


73 


Baldridge  National  Quality  Award.  The  application  process  for 
this  prestigious  award  is  generally  considered  to  be  a  very 
thorough  and  effective  evaluation  of  a  company's  quality 
attitudes,  values  and  assurance  procedures.  The  contents  of 
the  subordinate  subfactors  can  be  tailored,  or  entire 
subordinate  subfactors  added  or  deleted  depending  on  local 
procurement  activity  regulations  or  standard  operating 
procedures.  The  PRA  can  utilize  any  available  contractor 
historical  quality  data  collection  system  or  other  sources  of 
contractor  quality  data  such  as  described  in  Section  F  of 
Chapter  IV.  The  Army  has  proven  that  a  similar  source 
selection  procedure  can  be  defended  successfully  by  GAO  and 
the  Federal  District  Courts  as  long  as  the  contracting 
activity  conducts  the  evaluation  process  by  the  same  method 
that  it  describes  in  the  solicitation. 

Subsidiary  Research  Question  1:  How  are  Packard  Commission 


findings 

and  recommendations 

,  and  DoD-wide  TQM 

implementation  efforts  spawning 

attempts  to  incorporate 

gualitv 

considerations  into 

DoD  source  selection 

practices? 

After  the  Packard  Commission  performed  an  in-depth  study 
of  both  commercial  and  Government  procurement  practices  in 
198  6,  it  determined  that  DoD  procurement  has  a  tendency  to 
emphasize  award  to  the  lowest  priced  offer  at  the  expense  of 
other  important  objectives  such  as  quality,  delivery  and  low 
life  cycle  cost.  On  the  other  hand,  commercial  procurement 
officials  simultaneously  pursue  several  objectives:  attraction 
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of  the  best  qualified  suppliers,  validation  of  product 
performance  and  quality,  and  the  securing  of  the  best  price. 

To  remedy  this  situation,  one  of  the  Commission's 
recommendations  was  to  revise  Federal  Law  and  DoD  regulations 
to  provide  for  substantially  increased  use  of  commercial-style 
competition,  emphasizing  quality  and  established  performance 
as  well  as  price.  Price  should  not  be  the  sole  determinant, 
especially  for  procurement  of  complex  systems  and  services, 
according  to  the  Commission. 

There  are  numerous  other  efforts,  programs  and  regulations 
which  have  influenced  the  attitude  of  DoD  procurement 
activities  towards  quality  or  TQM  as  source  selection 
criteria.  They  range  from  an  Executive  Order  by  the  President 
of  the  United  States  to  the  use  of  TQM  and  Statistical  Process 
Control  factors  in  a  best  value  procurement  of  a  major  weapon 
system  by  the  Army  Communications-Electronics  Command.  A 
major  effort  to  develop  guidance  for  DoD  in  implementing  TQM 
throughout  not  only  the  defense  establishment  but  also  the 
defense  industrial  base,  began  with  the  Honorable  John  A. 
Betti's  formation  of  a  DoD/Joint  Service/  Agency  Process 
Action  Team.  This  team's  central  mission  was  to  develop  a 
guide  for  integrating  TQM  into  the  source  selection  process. 
However,  the  effort  has  not  progressed  beyond  a  second 
revision  of  the  Draft  Guide  due  to  Betti's  resignation. 

The  combination  of  the  desire  to  take  the  Draft  Guide  a 
few  steps  further,  and  of  the  success  of  the  ACEC  procurement 
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in  integrating  TQM,  SFQ  and  a  Performance  Risk  Assessment  into 
the  source  selection  process  has  formed  the  backbone  of  the 
model  source  selection  process  described  in  Chapter  III  of 


this  thesis. 

Subsidiary  Research  Question  2:  How  did  the  1984 
Competition  in  Contracting  Act  impact  PCO  use  of  best  or 
greatest  value  source  selection  procedures? 

Since  CICA  was  implemented  in  1984,  there  has  been  a 
widely  held  view  that  the  defense  procurement  emphasis  on  the 
importance  of  price  has  actually  intensified.  There  are  three 
reasons  for  this  based  according  to  the  Packard  Commission: 


1.  the  universal  interpretation  of  the  act's 
intentions  as  being  that  the  Government  must  buy 
from  the  lowest  bidder; 

2.  the  notion  among  contracting  activities  that  CICA 
precluded  ‘"he  use  of  oualif ication  criteria,  the 
co.nsiderat ion  of  technical  expertise  or  of  life 
cycxe  costs;  and 

3.  the  resulting  focus  on  the  numbers  of  competitions 
rather  chan  on  the  success  the  competition 
ac.tieves  in  term.s  of  better  products. 


The  goals  a.cd  intent  of  CICA  have  been  obscured  by  this 
n:r.  on  that  full  and  open  comipetition  precludes  best  or 
greatest  value  procureme.nos,  and  GAO  decisions  have  served 
only  to  reiiiforce  these  interpretations  and  .notions.  However, 
the  recent  passage  of  one  FY  1991  Authorization  Act  (P.L.  101- 
510)  has  resulted  in  a  revision  of  the  FAR,  and  a  removal  of 
the  restriction  on  award  without  discussion  in  instances  where 
the  award  does  not  result  in  the  lowest  overall  cost  to  the 
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Government.  This  reversal  of  the  impact  of  CICA  on  the  use  of 

best  value  procedures  on  DoD  acquisition  is  a  positive  step 

towards  mollifying  some  of  the  adverse  effects  of  CICA's 

implementation  and  interpretation. 

Subsidiary  Research  Question  3 :  What  are  the  current 
attitudes  of  Navy  PCO's,  PM' s  and  commercial  procurement 
personnel  towards  quality  factors  in  DoD  source  selection? 

CDR  Charles  A.  Perkins,  SC,  USN  performed  an  extensive 
survey  of  personal  interviews  with  Navy  PCO's,  Navy  Program 
Managers  (PM's)  and  members  of  the  National  Association  of 
Purchasing  Management  (NAPM)  in  1988  as  part  of  his 
dissertation  research.  Utilizing  questionnaires,  interviews, 
nonparametric  and  exploratory  data  analysis,  CDR  Perkins 
concluded : 


1.  price  competition  is  a  more  significant  factor  in 
Navy  procurement  than  in  commercial/industrial 
purchasing; 

2.  the  effectiveness  of  the  Navy  procurement  process 
in  obtaining  quality  products  is  inadequate 
because  of  the  over-emphasis  on  price  and  the  poor 
feedback  of  accurate  and  timely  qua]ity  related 
information  fromi  the  end-users  back  to  the  Navy 
PCO; 

3.  the  comparison  of  the  attitudes  and  preferences  of 
the  sample  populations  of  Navy  PCO' s  and  NAPM 
members  showed  a  significant  difference  in  the 
importance  that  they  place  on  quality  factors  in 
their  purchase  decisions,  with  heavier  attention 
placed  on  it  in  the  commercial/industrial  sector. 

4.  satisfaction  of  the  minimum  specifications  drives 
the  award  in  price-based  decisions;  and 

5.  consideration  of  other  non-price  related  factors 
evokes  a  notion  that  the  item  is  goldplated,  or 
that  too  m.uch  is  being  paid  for  an  item,. 
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Subsidiary  Research  Question  4:  What  crualitv  values  are 
considered  as  being  the  most  critical  and  ouantiifiabie  to 
Navy  PCO's  today? 

The  results  of  CDR  Perkin's  dissertation  research  also 
confirmed  that  there  is  a  definite  concurrence  among  Navy 


PCO' s. 

PM' s  and 

commercial 

procurement  personnel 

that 

performance  and 

reliability  are  the  numbers  one 

and 

two 

quality 

factors 

respectively 

in  their  ranking 

of 

the 

importance  of  quality  factors.  Navy  PCO' s  ranked  the  remaining 
quality  factors  in  order  of  performance  as  maintainability, 
schedule,  durability,  past  performance  and  warranty 
respectively.  Interviews  with  a  cross  section  of  Navy  PCO' s 
by  CDR  Perkins  also  confirmed  his  hypothesis  that  a  model 
method  of  source  selection  based  upon  prestated  quality 
measures  would  in  fact  be  utilized  in  some  of  their 
procurements  if  a  feasible  and  generally  acceptable  model  were 
available  to  them. 

Subsidiary  Research  Question  5:  What  are  the  various 
systems  throughout  DoD  which  collect  and  disseminate 
historical  contractor  crualitv  history  data,  and  could  be 
utilized  in  the  performance  of  a  PRA? 

The  following  is  a  brief  listing  of  the  most  current  data 
bases  designed  to  collect  and  disseminate  historical 
contractor  quality  history  data  (pure  application  systems  as 
opposed  to  data  collection  bases  are  not  listed) : 

NAVY 

1  .  Navy  Product  Deficiency  Reporting  and  Evaluation 
System  (PDREP) 
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ARMY 


2.  Army  Contractor  Information  System  (CIS) 

3 .  Army  Contractors  Requiring  Special  Attention 
(CRSA) 

4  .  Army  Commodity  Command  Standard  System  (CCSS) 

5 .  Army  Contractor  Performance  Certification  System 
(CP/2) 

6.  Army  Deficiency  Reporting  System  (DRS) 

7  .  Army  Contractor  History  Files 

AIR  FORCE 

8 .  Air  Force  Systems  Command  Contractor  Performance 
Assessment  Report  (CPAR) 

9  .  Air  Force _ Logistics _ Command  Contractor 

Responsibility  Review  Program  (CRRP) 

DEFENCE  LOGISTICS  AGENCY 

10  .  Defense  Logistics  Agency  Contractor  Profile  System 

1 1 .  DLA  Standard  Automated  Materiel  Management  System 
(SAMMS) 

12 .  DLA  Mechanization  of  Contract  Administration 
Services  (MOCAS) 

13.  DLA  Preaward  Survey  System  (PASS) 

14.  DLA  Customer /Depot  Complaint  System  (CDCS) 

15 .  DLA  Quality  Evaluation  Program 

E.  AREAS  FOR  FURTHER  RESEARCH 


1.  Talce  the  source  selection  model  and  send  it  to 
knowledgeable  and  experienced  purchasing  officials 
in  the  commercial  sector  for  their  input.  Modify 
and  refine  the  model  as  required  to  make  it  more 
feasible  for  actual  use  by  DoD  contracting 
activities . 
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2.  Take  the  model  and  persuade  a  DoD  contracting 
activity  to  use  the  model  in  a  pilot  procurement 
test.  Modify  and  refine  the  model  as  necessary 
based  on  the  results  of  this  pilot  test. 

3.  Perform  a  follow-on  study  of  the  Army 

Communicat ions-Electronics  Command  best  value 
procurement  of  third  generation  night  vision 
devices  at  a  $47M  premium,  to  determine  if  the 
premium  paid  was  cost  effective. 

4.  Compose  a  glossary  of  Total  Quality  Management  and 
other  quality  related  terms,  so  that  there  will  be 
standard  definitions  and  clear  terminology  for 
both  Government  and  industry  to  use  when 
contracting  for  best  value  and  quality. 
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APPEHDIX  A 

MALCOLM  BALDRIDGE  NATIONAL  QUALITY  AWARD 
EXAMINATION  GUIDELINES-  1990 


1 . 0  Leadership _ 

1.1  Senior  Executive  Leadership 

1.2  Quality  Values 

1.3  Management  for  Quality 

1.4  Public  Responsibility 

2 . 0  Information  and  Analysis _ 

2.1  Scope  and  Management  of  Quality  Data  and  Information 

2.2  Analysis  of  Quality  Data  and  Information 

3 . 0  Strategic  Quality  Planning 

3.1  Strategic  Quality  Planning  Process 

3.2  Quality  Leadership  Indicators  in  Planning 

3.3  Quality  Priorities 

4  ■  0  Human  Resource  Utilization 

4.1  Human  Resource  Management 

4.2  Employee  Involvement 

4.3  Quality  Education  and  Training 

4.4  Employee  Recognition  and  Performance  Measurement 

4.5  Employee  Well-Being  and  Morale 

5 . 0  Quality  Assurance  of  Products  and  Services 

5.1  Design  and  Introduction  of  Quality  Products  and  Services 

5.2  Process  and  Quc^’ty  Control 

5.3  Continuous  Improvement  of  Processes,  Products  and  Services 

5.4  Quality  Assessment 

5.5  Documentation 

5.6  Quality  Assurance,  Quality  Assessment  and  Quality 
Improvement  of  Support  Services  and  Business  Processes 

5.7  Quality  Assurance,  Quality  Assessment  and  Quality 
Improvement  of  Supplies 

6 . 0  Quality  Results 

6.1  Quality  of  Products  and  Services 

6.2  Comparison  of  Quality  Results 

6.3  Business  Process,  Operational  and  Support  Service 
Quality  Improvement 

6.4  Supplier  Quality  Improvement 
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7 . 0  Customer  Satisfaction 


7.1  Knowledge  of  Customer  Requirements  and  Expectations 

7.2  Customer  Realtionship  Management 

7.3  Customer  Service  Standards 

7.4  Commitment  to  Customers 

7.5  Complaint  Rsolution  for  Quality  Improvement 

7.6  Customer  Satisfaction  Determination 

7.7  Customer  Satisfaction  Results 

7.8  Customer  Satisfaction  Comparison 
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APPENDIX  B 

EXCERPT  FROM  THE  DoD/ JOINT  SERVICE /AGENCY  PAT: 
(DRAFT)  GUIDE  TO  INTEGRATING  TOTAL  QUALITY  MANAGEMENT 

INTO  SOURCE  SELECTION 


iii .  Process  for  Incorporating  Total  Quality  Management  in 
Source  Selection 


A.  Total  Quality  Management  in  the  Acquisition  Plan 
(AP)-The  AP  should  discuss  the  approach  suggested  in 
this  guide  to  incorporate  a  performance  risk 
evaluation  (using  Total  Quality  Management  and  past 
performance)  as  part  of  the  source  selection. 

B.  'T'otal  Quality  Management  in  the  Source  Selection 
Plan  (SSP)-  The  SSP  is  a  key  document  for  initiating 
and  conducting  source  selection.  It  is  prepared  by 
the  program  office  and  approved  by  the  SSA.  The  SSP 
should  provide  the  details  concerning  the  evaluation 
methodology  to  be  used  in  assessing  performance 
risk (refer  to  section  V  of  this  Guide  for  further 
information  on  conducting  the  assessment  of 
performance  risk)  .  The  SSP  should  also  stipulate  the 
specialized  training  requirements  of  the  performance 
risk  evaluators. 

IV .  Total  Quality  Management  in  the  Request  for  Proposals 

(RFP) 

This  section  provides  sample  language  for  use  in  the 
Executive  Summary  and  Sections  L  and  M  of  the  RFP. 

A.  Executive  Summary  of  the  RFP: 

1)  The  RFP  Executive  Summary  should  highlight  the 
Government's  intent  to  consider  continuous  process 
improvement  and  past  performance  as  an  overall 
performance  risk  assessment. 

2)  Sample  RFP  Executive  Summary  Language: 

"  The  Government  intends  to  consider  each 
offeror's  application  of  continuous  process 
improvement  principles  and  past  performance. 
Specifically,  a  performance  risk  assessment  will 
be  conducted  to  assess  the  effectiveness  of  the 
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offeror's  continuous  process  improvement  efforts 

and  past  performance." 

B.  Section  L-  Instructions  to  Offerors 

1  )  The  following  sample  instructions  to  offerors 
are  premised  on  the  Malcolm  Baldridge  Quality  Award 
Guidelines.  They  were  developed  by  tailoring  the 
Malcolm  Baldridge  criteria  in  scope  and  detail  and 
by  adding  DoD-oriented  language  to  make  the  criteria 
more  pertinent  for  use  in  DoD  source  selection.  This 
tailoring  preserves  the  basic  categories  and  depth 
of  coverage  needed  to  distinguish  between  the  levels 
of  mat-urity  of  continuous  process  improvement 
efforts  of  competing  offerors  and  to  specifically 
obtain  past  performance  information.  Source 
selection  personnel  should  be  encouraged  to  expand 
upon,  reduce  or  otherwise  tailor  the  language  as 
appropriate  to  their  specific  solicitation  needs. 
The  result  should  request  only  information  that  will 
support  the  performance  risk  assessment,  and  should 
not  require  Total  Quality  Management  plans  as 
contract  deliverables,  data  items,  etc.,  to  be 
developed  or  submitted. 

2)  The  Sample  RFP  Section  L  Language: 

"  Volume  XX-  Information  for  Assessment  of 

Performance  Risk,  Including  Continuous  Process 

Improvement  Efforts  and  Past  Performance 

Note :  The  use  of  a  separate  volume  is  required  to  obtain 

proper  visibility  and  to  focus  on  continuous  process 
Improvement  and  past  performance.  This  volume  may  be 
marked  "Source  Selection  Sensitive"  and  will  be 
treated  accordingly  by  the  Government . 

1.0  General.  The  Government  intends  to  consider 
each  offeror's  application  of  continuous  process 
improvement  principles  and  past  performance. 
Specifically,  a  performance  risk  assessment  will  be 
conducted  to  assess  the  effectiveness  of  the 
offeror's  continuous  process  improvement 
implementation  efforts  and  past  performance. 
However,  offerors  are  cautioned  that  in  conducting 
the  performance  risk  assessment,  the  Government  may 
use  data  provided  by  the  offeror  in  its  proposal  and 
data  obtained  from  other  sources. 

2.0  Specific  Information  and  Data.  The  offeror 
shall  provide  the  information  below  emphasizing 
documented,  verifiable  evidence  of  the  effective 
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implementation  of  continuous  process  improvement 
efforts  and  past  performance.  Actions  presently 
planned  shall  also  be  included.  Information 
provided  should  be  applicable  to  the  facilities  or 
location  where  work  under  the  proposed  contract  will 
be  performed.  The  offeror  shall  describe  the 
application  of  continuous  process  improvement 
practices,  tools  and  techniques,  and  shall  submit  a 
past  performance  record  on  relevant  contracts.  The 
offeror's  proposal  should  address  the  following 
areas : 

Leadership-  Describe  how  the  senior  executives 
create  and  sustain  a  clear  and  visible  quality  value 
system  along  with  a  supporting  continuous  process 
improvement  management  system  to  guide  all 
activities  of  the  company. 

Information  and  Analysis-  Describe  and  demonstrate 
the  scope,  validity,  and  management  of  data  and 
information  that  underlie  the  company's  continuous 
process  improvement  management  system.  In 
particular,  describe  how  the  company  uses  data  to 
support  a  prevention-based  approach  to  quality. 

Strategic  Quality  Planning-  Describe  the  company's 
continuous  process  improvement  priorities  and  plan 
to  achieve  them. 

Human  Resource  Utilization-  Describe  the  company's 
practices  to  develop  and  utilize  the  full  potential 
of  the  work  force  and  to  maintain  an  environment 
conducive  to  full  participation,  continuous  process 
improvement  and  personal  and  organizational  growth. 
Summarize  quantitative  and  qualitatively  1) 
accomplishments  to  date  and  recent  trends  in 
employee  participation  in  continuous  process 
improvement  activity,  2)  types  of  continuous  process 
improvement  education  and  training  provided  in  each 
pertinent  employment  category,  and  3)  trends  in 
recognizing  employees  for  contributions  to 
continuous  process  improvement. 

Quality  Assurance  of  Products  and  Services- 
Describe  how  products  and  services  are  continuously 
improved  through  optimization  and  improvement  of 
processes.  In  the  area  of  design  and  development, 
where  applicable,  the  description  may  include 
information  pertaining  to  the  use  of  continuous 
process  improvement  techniques  such  as  Quality 
Function  Deployment,  producibility  engineering  and 
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planning,  design  of  experiments,  DoD  Directive 
4245. 7M  -  Transition  from  Development  to  Production, 
etc.  Include  a  description  of  how  the  offeror  flows 
the  company' s  continuous  process  improvement  focus 
down  to  subcontractor  levels. 

Results  -  Provide  data  that  shows  trends  in: 
improvement  of  quality  of  products  and  services 
based  on  analysis  of  customer  requirements,  analysis 
of  quality  deficiency  reports,  cycle  t^me 
reductions.  Material  Review  Board  actions,  scrap  and 
rework,  etc.,  and  the  analysis  of  internal  business 
operations,  and  improvement  in  the  quality  of 
supplies  and  services  furnished  by  other  companies. 
Provide  evidence  of  the  use  of  results  to  overcome 
and  prevent  the  recurrence  of  problems.  Demonstrate 
application  of  the  offeror's  continuous  process 
improvement  activities  by  briefly  summarizing 
several  projects  that  illustrate  their  breadth  and 
effectiveness.  The  offeror  should  submit 

information  on  contracts  considered  relevant  in 
demonstrating  ability  to  perform  on  the  proposed 
effort.  This  information  may  include  data  on  efforts 
performed  by  other  divisions,  corporate  management, 
critical  subcontractors,  or  teaming  subcontractors, 
if  such  resources  will  be  brought  to  bear  or 
significantly  influence  the  p^^rformance  of  the 
proposed  effort.  Fcr  each  current  or  past  (within 
the  prior  three  years)  contract  which  is  relevant 
(similar  etfoit  of  for  items  of  comparable 
complexity) ,  the  offeror  should  provide  the 
following  information: 

a)  Admi-.istrative  Data 

(i)  Company/Division  Name 

(ii)  Program  Title 

(iii)  Contracting  Agency 

viv)  Contract  Number  and  Award  Date 

(v)  A  Brief  D'^^script  ion  of  the  Contract 
Effort,  Indicating  Whether  it  Was 
Development  and/or  Production 

(vi)  Type  of  Contract 

(vii)  Period  of  Performance 

(viii)  Original  Contract  Dollar  value  and 

Current  Dollar  Value 

(ix)  Original  and  Current  Schedules  and 
Completion  Dates 

(x)  Name,  Address,  and  Telephone  Number  of 
Government  Program  Director/Manager , 
Administrative  Contracting  Officer,  and 
PCO. 


86 


b)  Specific  Content-  Offerors  are  required  to 
explain  aspects  of  the  contracts  that  are  deemed 
relevant  to  the  proposed  effort.  Offerors  are  also 
requested  to  submit  information  on  significant 
achievements  or  explain  past  problems  they  consider 
relevant  to  the  proposed  effort. 

Customer  Satisfaction-  Describe  the  company' s 
methods  for  determining  customer  satisfaction. 
Briefly  summarize  the  company's  customer 
satisfaction  trends." 


C.  Section  M  -  Evaluation  Factors  for  Award 

1)  The  language  for  Section  M  should  be  Covered  in 
two  Parts: 

a)  General  Basis  for  Award  -  This  section 
describes  how  the  solicited  proposals  will 
be  evaluated  and  rated. 

b)  Performance  risk  -  This  section  describes 
how  performance  risk  will  be  assessed. 

2)  Sample  RFP  Performance  Risk  Language: 

The  Government  will  also  conduct  a  performance  risk 
assessment  based  upon:  1)  effectiveness  of  the 

offeror's  continuous  process  improvements  effort  and 
the  applicability  of  the  offeror's  use  on  continuous 
process  improvement  practices,  tools,  and 
techniques,  and  2)  the  offeror's  past  performance 
record,  demonstrated  in  terms  of  actual  results.  In 
assessing  performance  risk,  the  Government  may  use 
information  and  data  as  submitted  by  the  offeror,  as 
well  as  information  and  data  obtained  from  other 
sources,  to  evaluate  the  areas  listed  above. 
Performance  risk  assessment  relates  to  the  offeror's 
probability  of  successfully  accomplishing  the 
proposed  effort." 
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EVALUATION  FORM 


FACTOR: 


SUBFACTOR :  Leadership 


SUBORDINATE  SUBFACTOR:  A  (2)  Management  for  Quality 

Evaluate  how  the  company  integrates  its  quality  values  into  day-to 
day  management  of  all  units.  Examples  may  include  but  are  not  limited  to 

-  Key  strategies  for  invol¬ 
ving  all  levels  of  manage¬ 
ment  and  supervision  in  the 
monitoring  and  promotion  of 
quality,  and  the  principal 
roles  and  responsibilities 
defined  at  each  level. 

-  Key  strategies  to  promote 
cooperation  and  synergism 
among  managers  and  supervi¬ 
sors  of  all  units  at  all 
levels.  Examples  include: 

*  interunit  teams 

*  process  action  teams 
-Types,  frequency  and  con¬ 
tent  of  company  reviews  of 
the  status  of  quality  plans 
and  actions  to  assist  units 
not  performing  up  to  poten¬ 
tial  . 


Maximum 

Points 

Assessment 

O. 

Score 

Scoring  Justification: 

1.  Strong  Points. 

2 .  Weak  Points  . 

3.  Overall  Rationale  for  Assessment  %. 
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EVALUATION  FORM 


FACTOR: 


SUBFACTOR :  Information  and  Analysis 


Maximum 

Points 


Assessment 


Score 


SUBORDINATE  SUBFACTOR:  B  (1)  Scope  and  Management  of  Quality  Data  and 
Information . 

Evaluate  the  adequacy  of  the  company's  management  information 
system  to  support  the  planning,  management  and  evaluation  of  qual-ity,  and 
how  data  and  information  reliability,  timeliness  and  access  are  assured. 
Examples  may  include  but  are  not  limited  to: 

Documents/facts  verifying  the  existence  of — 

-  Criteria  for 
selection  of  specific  items 
for  inclusion  in  the  qual¬ 
ity  related  data  and  infor- 
ation  base. 

-  A  valid  and  effective  use 
of  the  data,  including  int¬ 
ernal  operations  and  proc¬ 
esses,  quality  results,  su¬ 
pplier  quality,  etc. 

-  Processes  and 
technologies  the  company 
uses  to  ensure  the  validity 
consistency, standardization 
review, update  and  timely 
access  to  quality  data 
throughout  the  company. 


Scoring  Justification: 

1.  Strong  Points. 

2.  Weak  Points. 

3.  Overall  Rationale  for  Assessment  %. 
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EVALUATION  FORM 


FACTOR: 


SUBFACTOR :  Information  and  Analysis 


Maximum 

Points 


Assessment 


Score 


SUBORDINATE  SUBFACTOR:  B  (2)  Analysis  of  Quality  Data  and  Information 

Evaluate  how  data  and  information  are  systematically  collected  and 
analyzed  to  support  the  company's  quality  objectives 

in  a  timely  and  efficient  manner.  Examples  may  include  but  are  not  limited 
to : 

Documents/facts  verifying  the  existence  of —  - 

Principal  types  of  anal¬ 
ysis  performed  such  as  det¬ 
ermination  of  trends  and 
projections  of  quality  imp¬ 
rovements  that  should  res¬ 
ult  from  changes  in  techno¬ 
logy,  evaluation  of  system 
performance,  and  assessment 
of  product  long-term  perfor 
mance . 

-  Plans  made 

and  steps  taken  to  shorten 
the  cycle  of  data  gathering 
analysis, and  access  to  imp¬ 
rove  support  of  company 
quality  objectives. 

-  Analysis  perfromed 
leads  to  changes  in  types 
of  data  collected,  improved 
reliability  of  data, and  im¬ 
proved  analytical  capabil¬ 
ities  . 


Scoring  Justification: 

1.  Strong  Points. 

2.  Weak  Points. 

3.  Overall  Rationale  for  Assessment  %. 
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EVALUATION  FORM 


FACTOR: 


SUBFACTOR:  Strategic  Quality  Planning 


SUBORDINATE  SUBFACTOR:  C  (1)  Strategic  Quality  Planning  Process 

Evaluate  the  company's  strategic  quality  planning  process  for 
short  term  (1-2  years)  and  long  term  (3-5  years)  to  achieve  and 
sustain  leadership  and  customer  satisfaction 

Examples  may  include  but  are  not  limited  to:  Documents/facts  verifying  the 
existence  of — 

-  Employees, 

suppliers  and  customers  con 
tributing  to  the  planning 
process,  including  data  co¬ 
llection  and  analysis. 

-  The  company  attempts  to 
integrate  short  and  long 
term  priorities  into  the  co 
mpany' s  leadership  object¬ 
ives  . 

-  There  are  mechanisms  by 
which  the  company  ensures 
that  its  suppliers  meet  or 
exceed  its  quality  require¬ 
ments  . 

-  The  company  has  attempted 
to  perform  a  self-diagnosis 
or  self  assessment  of 
its  quality  program  such  as 
through  use  of  the  Malcolm 
Baldridge  National  Quality 
Award  application  guideline 
as  evaluation  criteria. 

-  The  planning 

process  is  continuously  evaluated  and  improved. 

Scoring  Justification: 


Maximum 

Points 

Assessment 

% 

Score 

1.  Strong  Points. 

2.  Wealc  Points. 

3.  Overall  Rationale  for  Assessment  %. 
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EVALUATION  FORM 


FACTOR : 


SUBFACTOR :  Human  Resource  Utilization 


SUBORDINATE  SUBFACTOR:  D  (1)  Employee  Involvement 

Evaluate  how  the  company  provides  a  means  for  all  of  its  employees  to 
effectively  contribute  to  the  company's  quality  objectives.  Examples 
include  but  are  not  limited  to;  Documents/ 
facts  verifying  the  existence  of — 

-  Company  encouragement 
of  group  participation 
to  evaluate  and 
assist  in  redesign  of  the 
company' s  systems  and  proc¬ 
esses,  examples  may  include 

*  Steering  groups 

*  Improvement  teams 

*  Problem-solving 
teams 

*  Quality  circles 
-Other  avenues  for  employee 
participation,  including 
suggestion  systems,  ombud¬ 
smen,  or  hotlines. 

-  The  company  attempts  to 
enhance  employee  power  to 
act  (empowerment) . 

-  The  company 

utilizes  surveys,  question¬ 
naires  or  other  methods  to 
collect  information  and  to 
provide  a  baseline  measure¬ 
ment  of  employee  attitudes  and  values. 

-  The  company  provides  prompt,  constructive  feedback  on  and  rewards  for 
employee  suggestion,  comments  and  criticisms. 

Scoring  Justification: 

1.  Strong  Points. 

2.  Weak  Points. 


3.  Overall  Rationale  for  Assessment  %. 


Maximum 

Points 

Assessment 

a 

c 

Score 
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EVALUATIOM  FORM 


FACTOR : 


SUBFACTOR :  Human  Resource  Utilization 


Maximum 

Points 


Assessment 


Score 


SUBORDINATE  SUBFACTOR:  D  (2)  Quality  Education  and  Training 

Evaluate  how  the  company  provides  quality  related  education  and 
training  for  employees , and  how  it  subsequently  utilizes  the 
knowledge  and  skills  acquired  in  the  process.  Evaluate  how  the  company 
measures  human  resource  contributions  so  that  management  can  make  informed 
decisions  about  training,  recognition  and  employee  involvement.  Examples 
may  include  but  are  not  limited  to: 

Documents/facts  verifying  the  existence  of — 

-  Company  rationale 
for  deciding  what  education 
and  training  is  needed 
for  different  categories 
of  employees. 

-  The  company 

provides  on-the-job  reinfo¬ 
rcement  of  the  knowledge 
and  skills  acquired  in  edu¬ 
cation  and  training. 

-  Indicators  of  effective¬ 
ness  of  the  company' s  edu¬ 
cation  and  training  activ¬ 
ities  and  how  the  indicat¬ 
ors  are  used  to  improve 
these  activities. 

-  Company  ties  needs-based 
training  with  business 
plans . 

-  Company  designs 
results/benefits  measure¬ 
ment  into  the  training  proc 
ess . 


t 


t 


Scoring  Justi.  ::rtion: 

1 .  Strong  Points . 

2.  Weak  Points. 

3.  Overall  Rationale  for  Assessment  %. 
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EVALUATION  FORM 


FACTOR: 


SUBFACTOR :  Human  Resource  Utilization 


SUBORDINATE  SUBFACTOR:  D  (3)  Employee  Recognition  and  Performance 
Measurement 

Evaluate  how  the  company's  employee  recognition  and  performance 
measurement  process  supports  quality  improvement. 

Examples  may  include  but  are  not  limited  to: 

Documents/facts  verifying  the  existence  of — 


-  Company  strategies  for  en 
couraging  contributions  to 
quality,  including  recogni¬ 
tion  of  individuals  or 
groups,  and  suppliers  for 
exemplifying  quality. 

-  The  company 

evaluates  the  effectiveness 
of  its  recognition  and  per¬ 
formance  measurement  system 
including  mechanisms  to 
solicit  employee  feedback. 

-  The  company  provides 
prompt,  constructive  feed¬ 
back  on  and  rewards  for 
employee  suggestions, 
comments,  criticisms. 


Maximum 

Points 

Assessment 

g, 

c 

Score 

Scoring  Justification: 

1.  Strong  Points. 

2.  Weak  Points. 


3.  Overall  Rationale  for  Assessment  %. 


EVALUATION  FORM 


FACTOR : 


SUBFACTOR :  Human  Resource  Utilization 


SUBORDINATE  SUBFACTOR:  D  (4)  Employee  Well-Being  and  Morale 

Evaluate  how  the  company  safeguards  the  health  and  safety  of 
employees,  ensures  their  comfort  and  physical  protection,  and  maintains  a 
supportive  work  environment.  Examples  may  include  but  are  not  limnited  to: 
Documentation/facts  verifying  the  existence  of — 

-  The  company 
integrates  well  being  and 
morale  factors  such  as 
health,  safety,  job  satis¬ 
faction  and  ergonomics  into 
quality  improvement  activ¬ 
ities.  Examples  may  include 

*  Wellness  classes 

*  Special  health  events 

*  Health  risk  assess¬ 
ments  for  employees 
and  their  families 

-  The  quality  and  effectiv¬ 
eness  of  the  company' s  meth 
ods  for  analysis  of  under¬ 
lying  causes  of  accidents, 
work  related  health  prob¬ 
lems  and  job  dissatisfact¬ 
ion,  and  the  elimination  of 
adverse  conditions. 

-  Special  services,  facil¬ 
ities  and  opportunities  the 
company  makes  available  to 
support  employees. 

Scoring  Justification: 

1.  Strong  Points. 

2.  Weak  Points. 


Maximum 

Points 

Assessment 

o 

•6 

Score 

o 

o  • 


3.  Overall  Rationale  for  Assessment 


EVALUATION  FORM 


FACTOR: 


SUBFACTOR :  Quality  Assurance  of  Products  and  Services 


Maximum 

Points 


Assessment 


score 


SUBORDINATE  SUBFACTOR:  E  (1)  Design  and  Introduction  of  Quality 
Products  and  Services 

Evaluate  the  process  by  which  new  or  improved  products  and  services 
are  designed  and  introduced  to  meet  or  exceed  customer  requirements,  and 
how  processes  are  designed  to  deliver  according  to  the  requirements. 
Examples  may  include  but  are  not  limited  to: 

Documentation/facts  verifying  the  existence  of: 

-  Effective  mechanisms  for 
conversion  of  customer 
needs  and  expectations  into 
product  and  process  require 
ments  and/or  service  qual¬ 
ity  standards. 

-  The  company's  methods  for 
assuring  that  quality  is  de 
signed-in  during  research, 
development,  and  validation 
stages,  including  review  of 
designs  for  feasibility, and 
assessment  of  key  factors 
in  production  and  use. 

-  The  company 

has  a  detailed  control  plan 
including  provisions  for 
the  selection  and  setting 
of  key  products  character¬ 
istics  to  be  controlled  and 
how  they  are  controlled. 


Scoring  Justification: 

1.  Strong  Points. 

2.  Weak  Points. 

3.  Overall  Rationale  for  Assessment  %. 
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EVALUATION  FORM 


FACTOR : 


SUBFACTOR:  Quality  Assurance  of  Products  and  Services 


SUBORDINATE  SUBFACTOR:  E  (2)  Process  and  Quality  Control 

Evaluate  the  processes  by  which  the  company' s  products  or  services 
are  controlled,  and  how  the  company  assures  that  products  or  services  meet 
design  plans,  specifications  or  SOW' s .  Examples 

may  include  but  are  not  limited  to:  Documentation/facts  verifying 
the  existence  of — 

-  Effective  approaches  the 
company  uses  to  ensure  that 
processes  which  produce  pro 
ducts  or  services  are  adequ 
ately  controlled  to  ensure 
quality,  timely  delivery 
within  cost  objectives. 

-  Effective  approaches  the 
company  takes  to  ensure 
that  products  and  services 
meet  design  plans,  specs  or 
SOW' s . 

-  Effective  approaches  the 
company  uses  to  identify 
root  causes  of  process  up¬ 
sets  (problems) . 

-  Effective  approaches  to 
the  design  of  the  measures 
to  correct  process  upsets, 
and  methods  of  verifying 
that  the  measures  produce 
the  predicted  results,  and 
are  effectively  utilized  in 
all  appropriate  units  of 
the  company. 

-  Effective  approaches  the  company  uses  to  utilize  the  information 
obtained  from  process  and  quality  control  for  problem  prevention  and 
quality  improvement. 

Scoring  Justification: 

1.  Strong  Points. 

2 .  Weak  Points . 


3.  Overall  Rationale  for  Assessment  %. 


Maximum 

Points 

Assessment 

o 

o 

Score 
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EVALUATIOK  FORM 


FACTOR: 


SUBFACTOR :  Quality  Assurance  of  Products  and  Services 


SUBORDINATE  SUBFACTOR:  E  (3)  Continuous  Process  Improvement  of  Processes, 
Products  and  Services 

Evaluate  how  the  company's  products/services  are  continuously 
improved  through  optimization  and  improvement  processes.  Examples 
may  include  but  are  not  limited  to: 

Documentation/facts  verifying  the  existence  of — 

-  The  company 
utilizes  methods  for  the 
continuous  improvement  of 
quality  in  products/service 
using  any  of  the  following 
techniques  (or  any  others 
which  the  offeror  presents) 

*  Cause  and  Effect  Analysis 

*  Pareto  Charts 

*  Statistical  Process  Cont¬ 
rol  (SPC) 

*  Histograms 

*  Checksheets 

*  Input/Output  Analysis 

*  Scatter  Diagrams 

*  Concurrent  Engineering 

*  Control  Charts 

*  Work  Flow  Analysis 

*  Shewart  Cycles 

-  The  company 

has  a  method  of  integrating 
continuous  process  improve¬ 
ment  into  daily  operations 
and  routine  process  and 
quality  control  by  all  un¬ 
its  of  the  company. 

Scoring  Justification: 

1.  Strong  Points. 

2.  Weak  Points. 


3.  Overall  Rationale  for  Assessment  %. 


Maximum 

Points 

Assessment 

o 

c 

Score 
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E VALUATION  FORM 


FACTOR 

SUBFACTQR :  Quality  Assurance  of  Products  and  Services 


SUBORDINATE  SUBFACTOR:  E  (4)  Quality  Assurance,  Quality  Assessment  and 
Quality  Improvement  of  Suppliers 

Evaluate  how  the  quality  of  materials,  components  and  services 
furnished  by  other  businesses  is  assured  and  continuously  improvob . 
Examples  may  include  but  are  not  limited  to: 

Documentat ion/ facts  verifying  the  existence  ot — 

-  Effective  processes  used  ir -  I 

to  ensure  that  the  company  Maximum  Assessment  Score 
quality  requirements  are  Points  % 

being  met  by  suppliers,  - - 

through  such  means  as: 
audits,  inspections,  cert¬ 
ification  and  ter.Jng,  part 
nerships ,  training,  cont¬ 
ract  incentives  and  recog¬ 
nition  . 


Scoring  Justification ; 


1.  Strong  Points. 


2.  Weak  Points. 


3.  Overall  Rationale  for  Assessment 


o 

o  • 


EVALUATION  FORM 


FACTOR: 


SUBFACTOR :  Customer  Satisfaction 


SUBORDINATE  SUBFACTOR:  F  (1)  Knowledge  of  Customer  Requirements  and 
Expectations 

Evaluate  how  the  company  effectively  manages  its  relationships  with 
customers,  broken  out  by  commercial/Government  categories,  and  how  it 
ensures  continuous  improvements  of  customer  relationship  management. 
Examples  may  include  but  are  not  limited  to; 

Documentat ion/ facts  supporting  the  existence  of — 

-  Effective  means  for  ens¬ 
uring  easy  access  for  cust¬ 
omers  to  comment,  seek  ass¬ 
istance  or  to  complain. 

-  Company  follow-ups 
with  customers  on  comments, 
complaints,  requests  for 
assistance . 

-  Mechanisms  in 
place  to  empower  customer- 
contact  employees  to  take 
e;;traordinary  measures  to 
accommodate  the  customer 
when  necessary. 

-  The  use  of  te..hnology  and 
logistics  (infrastructure) 
support  to  enable  customer- 
contact  employees  to  prov¬ 
ide  effective  and  timely 
customer  service. 

-  Company  analysis  of 
complaint  inf ormation, gains 
and  losses  of  customers , and 
lost  orders  to  assess  costs 
and  market  consequences  for 

Scoring  Justification: 

1.  Strong  Points. 

2 .  Weak  Points . 


3.  Overall  Rationale  for  Assessment  %. 


1  Maximum 
Points 

Assessment 

Score 

policy  review. 
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EVALUATION  FORM 


FACTOR : 


SUBFACTOR :  Customer  Satisfaction 


Maximum 

Points 


Assessment 


Score 


SUBORDINATE  SUBFACTOR:  F  (2)  Customer  Service  Standards 

Evaluate  the  company's  standards  governing  the  direct 
between  employees  and  customers,  and  how  these  standards  are 
modified.  Examples  may  include  but  are  not  limited  to: 
Documentation/facts  supporting  the  existence  of — 

-  The  existence  of  product 
and  service  guarantees  and 
product  warranties,  and 
other  types  of  commitments 
the  company  makes  to  pro¬ 
mote  trust  and  confidence 
in  its  products  and  serv¬ 
ices  .  Evaluate  the  compre¬ 
hensiveness,  conditional 
qualifications,  understand- 
ability  and  credibility. 

-  How  improvements  in  the 
company's  products/services 
over  the  past  3  years  have 
been  translated  into  change 
in  warranties,  guarantee's 
other  commitments. 


contact 
set  and 


Scoring  Justification: 

1.  Strong  Points. 

2.  Weak  Points. 


3.  Overall  Rationale  for  Assessment  %. 
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APPENDIX  D 
EVALUATION  FORM 


( 


(EXAMPLE) 

Basic  Proposal  Evaluation 
XYZ  Corporation  Proposal 

FACTOR:  Continuous  Process  Control 

SUBFACTOR :  Quality  Assurance  of  Products  and  Services 

SUBORDINATE  SUBFACTOR:  E  (3)  Continuous  Process  Improvement  of  processes, 
Products  and  Services 

Evaluate  how  the  company's  products/services  are  continuously 
improved  through  optimization  and  improvement  processes.  Examples: 

-  Evidence  that  the  company 
utilizes  methods  for  the 
coninuous  improvement  of 
quality  in  products/service 
using  any  of  the  following 
techniques : 

*  Cause  and  Effect  Analysis 

*  Pareto  Charts 

*  Statistical  Process  Cont¬ 
rol  (SPC) 

*  Histograms 

*  Checksheets 

*  Input /Output  Analysis 

*  Scatter  Diagrams 

*  Concurrent  Engineering 

*  Control  Charts 

*  Work  Flow  Analysis 

*  Shewart  Cycles 

-  Evidence  that  the  company 
has  a  method  of  integrating 
continuous  process  improve¬ 
ment  into  daily  operations 
and  routine  process  and 
qualify  control  by  all  un¬ 
its  of  the  company. 

Scoring  Justification: 

1.  Strong  Points.  XYZ  Corporation's  proposal  is  very  detailed  and  specific 
with  regard  to  its  successful  use  of  SPC  to  control  the  widget  production 
process.  Its  description  of  the  Measurement  Error 

study, Control  Charting, Cpk  Capability  Index,  Taguchi  Loss  Function 
control  processes,  and  SPC  applications  to  assemblies,  subassem-blies  and 
components  show  positive  trends  in  continuous  process 
improvements  as  evidenced  by  a  4%  drop  in  scrap  rate  last  year. 

2.  Weak  Points.  Not  all  of  XYZ  Corporation's  divisions,  affiliates  and 
suppliers  have  this  successful  SPC  and  continuous  process  improvement 
program  in  place,  or  they  are  in  place,  but  not  as  mature  or  successful  as 
they  are  for  the  widget  production  process. 

3.  Overall  Rationale  for  Assessment  %:  This  is  clearly  a  "superior" 
component  of  the  subfactor,  but  not  100%  due  to  the  weak  point. 


Maximum 

Points 

Assessment 

o 

c 

Score 

100 

90% 

90 
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EVALUATION  FORM 

(EXAMPLE) 

Peformance  Risk  Assessment 
XYZ  Corporation 

FACTOR :  Continuous  Process  Control 

SUBFACTOR :  Quality  Assurance  of  Products  and  Services 

SUBORDINATE  SUBFACTOR:  E  (3)  Continuous  Process  Improvement  of  processes. 
Products  and  Services 

Evaluate  how  the  company's  products/services  are  continuously 
improved  through  optimization  and  improvement  processes.  Examples: 
Evidence  that  the  company 
utilizes  methods  for  the 
coninuous  improvement  of 
quality  in  products/service 
using  any  of  the  following 
techniques : 

*  Cause  and  Effect  Analysis 

*  Pareto  Charts 

*  Statistical  Process  Cont¬ 
rol  (SPC) 

*  Histograms 

*  Checksheets 

*  Input/Output  Analysis 

*  Scatter  Diagrams 

*  Concurrent  Engineering 

*  Control  Charts 

*  Work  Flow  Analysis 

*  Shewart  Cycles 
-  Evidence  that  the  company 
has  a  method  of  integrating 
continuous  process  improve¬ 
ment  into  daily  operations 
and  routine  process  and 
quality  control  by  all  un¬ 
its  of  the  company. 

Scoring  Justification: 

1.  Strong  Points:  A  review  of  PDREP  database,  preaward  surveys  and  on-site 
plant  visits  performed  in  conjunction  with  two  Navy  production  contracts 
within  the  past  two  years  for  a  wodget,  similar  in  most  respects  to  a 
widget,  revealed  that  XYZ's  SPC  and  continuous  improvement  processes  were 
adequate,  with  some  weaknesses  noted. 

2.  Weak  Points:  A  review  of  the  applicable  data  in  the  PDREP  data  base, 
preaward  surveys  and  results  of  on-site  plant  visits  reveals  notable 
deficiencies  in  XYZ  Corporation's  SPC  procedures,  espec-ally  with  regard 
to  the  machined  parts  provided  by  a  supplier,  and 

in  XYZ's  manufacture  of  Vacuum  Tube  Envelopes  and  microchannel  plates. 
Further,  it  was  determined  that  XYZ  was  not  making  any  significant  effort 
to  incentivize  its  supplier  to  use  SPC  in  its  process,  nor  was  it  making 
an  effort  to  locate  a  better  supplier. 

3.  Overall  rationale  for  Aasessment  %:  In  view  of  how  critical  the 
machined  parts.  Vacuum  tube  envelopes  and  microchannel  plates  are  in  the 
proper  functioning  of  widgets,  and  how  little  progress  XYZ  has  made  to 
correct,  the  risk  in  this  area  is  high-moderate. 


Maximum 

Points 

Assessment 

c. 

Score 

100 

70% 

70 

I 


/ 


I 


* 
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APPENDIX  E 


Evaluator 

Date : _ 

RFP : _ 

CONSOLIDATED  TALLY  SHEET 
(EXAMPLE) 

Basic  Proposal  Evaluation 


Factors /Subfactors 


Max 

Points 


Offerors 


20 

30 

20 

30 

100 


70 


70 


Leadership  and  Personnel  Management 

A.  Leadership 

(1)  Senior  Executive  Leadership 

(2)  Management  For  Quality 

B.  Information  and  Analysis 

(1)  Scope  and  Management  of  Data 

(2)  Analysis  of  Quality  Data 

C.  Strategic  Quality  Planning 
(1)  Strategic  Quality  Planning 

Process 

D.  Human  Resource  Utilization 

(1)  Employee  Involvement 

(2)  Quality  Education  and 
Training 

(3)  Employee  Recognition  and 

Performance  Measurement  60 

Factor  Totals  400 

Continuous  Process  Improvement 

E.  Quality  Assurance  of  Products 
and  Services 

(1) Design  and  Introduction  of  100 

Quality  Products/Services 

(2)  Process  and  Quality  Control  100 

(3)  Continuous  Process  Improvement  100 
of  Processes,  Products/Services 

(4)  Quality  Assurance, Quality  Ass-  100 
essments  and  Quality  Improve-Suppliers 

F.  Customer  Satisfaction 

(1)  Knowledge  of  Customer  Require¬ 
ments  and  Expectations  100 

(2)  Customer  Service  Standards  100 


18 

27 

18 

27 

90 


63 

63 

54 


360 


90 

90 

90 

90 


90 

90 


Factor  Totals 


600 


540 


105 


Evaluator 
Date : 

(EXAMPLE)  RFP:_^ 

Consolidated  Tally  Sheet 
Performance  Risk  Assessment 


Factors /Subfactors 


Max 

Points 


Offerors 


Leadership  and  Personnel  Management 

A.  Leadership 

(1)  Senior  Executive  Leadership 

(2)  Management  For  Quality 

B.  Information  and  Analysis 

(1)  Scope  and  Management  of  Data 

(2)  Analysis  of  Quality  Data 

C.  Strategic  Quality  Planning 
(1)  Strategic  Quality  Planning 

Process 

D.  Human  Resource  Utilization 

(1)  Employee  Involvement 

(2)  Quality  Education  and 
Training 

(3)  Employee  Recognition  and 
Performance  Measurement 

Factor  Totals 

Continuous  Process  Improvement 

E.  Quality  Assurance  of  Products 
and  Services 

(1) Design  and  Introduction  of 
Quality  Products/Services 

(2)  Process  and  Quality  Control 

(3)  Continuous  Process  Improvement 
of  Processes,  Products/Services 

(4)  Quality  Assurance, Quality  Ass¬ 
essments  and  Quality  Improve-Suppliers 

F.  Customer  Satisfaction 

(1)  Knowledge  of  Customer  Require¬ 
ments  and  Expectations  100 

(2)  Customer  Service  Standards  100 


20 

30 

20 

30 

100 


70 


70 


60 

400 


100 

100 

100 

100 


14 

21 

14 

21 

70 


19 

49 

42 


250 


70 

70 

70 

70 


70 

70 


Factor  Totals 


600 


420 
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AN  EXAMPLE  OF  SSEB  CONVERSION  OF  SCORES  TO  ADJECTIVAL  RATINGS /RISKS 


A.  Example  Conversion  Scales 


Basic  Proposal  Evaluation  Scale:  i 

PRA  Risk  Level  Scale: 

Risk 

Factor : 

Point  Range: 

Rating: 

Point  Range 

:  Level 

Continuous 

540-600 

Superior 

510-600 

Low 

Process 

420-539 

Acceptable 

420-509 

Moderate 

Improvement 

360-419 

Marginal 

0-419 

High 

0-359 

Unsat 

Leadership 

and  360-400 

Superior 

340-400 

Low 

Personnel 

280-359 

Acceptable 

280-339 

Moderate 

Management 

240-279 

Marginal 

0-279 

High 

0-239 

Unsat 

B.  Example 

Conversion  of  XYZ 

Corporation's  Scores  by 

SSEB: 

Risk 

Factor : 

Points : 

Rating : 

Points : 

Level : 

CPI 

540 

Superior 

420 

Moderate 

LPM 

360 

Superior 

250 

High 

C.  An  example  determination  of  Overall  Factor  Rating  by  SSEB: 


Factor : 


Basic 

Evaluation 

Rating: 


Performance 

Risk 

Assessment : 


Overall 

Rating: 


CPI 

LPM 


Superior 

Superior 


Moderate 

High 


Acceptable 

Marginal 


A 
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